VictorOps - SolarWinds Integration

Version 3

    Overview

    The VictorOps Ack-Back Application Template allows you to acknowledge and resolve alerts from the VictorOps portal. The application template consists of a single Windows PowerShell Script monitor component. The Powershell script queries the VictorOps public API to get a current list of acknowledged and resolved incidents.

     

    Environments Tested With

    SAM 6.2.4, NPM 12.0, SolarWinds SDK 2.1.13

     

    Requirements

    If you do not own SAM it is possible to run the same powershell script using a windows scheduled task. You will need to update the SWIS credentials in the script manually.

    #Create SWIS connection object

    $Swis = Connect-Swis –Hostname $SolarWindsServer -UserName “solarwindsapi" -Password "password" -IgnoreSslErrors

     

    Known Limitations and Issues

    • You cannot use a \ or : in the VictorOps HTTP post. You will need to use an SQL replace function like below in the SolarWinds Alert.
      SQL: SELECT REPLACE ('''${Caption}''','\',' ')}
    • There may be other special characters not supported with the SolarWinds HTTP post not documented here.
    • You must include the SolarWinds variable ${N=Alerting;M=AlertObjectID} in the HTTP Post in your Alert. This contains the alert’s unique ID.
      Note – this variable is new to NPM 12 and SAM 6.2.4
    • The VictorOps Incidents API does not return any custom fields you may be sending in your HTTP Post
    • Unable to set the AcknowledgedBy field. This will always show the service account running the script.

     

    Setup and Configuration

     

    Create a Limited Rights user for SolarWinds Alert Management

    1. From the Main Settings & Administration section in SolarWinds go to User Accounts > Manage Accounts
    2. Select ADD NEW ACCOUNT
    3. Select Orion individual account, click Next
    4. In the User Name field enter solarwindsapi
    5. Enter a unique strong password, click Next
    6. In the Define Settings section scroll down to ALERTS. Change the dropdown box Allow Alert Management Rights to Yes
    7. Scroll to the bottom and Click Submit

    Add the Limited Rights user to the SAM Credential Library

    1. From the Main Settings & Administration section in SolarWinds go to SAM Settings > GLOBAL SAM SETTINGS > Credential Library
    2. Add the limited user account created above. For the credential name use SolarWinds API User
    3. From the Manage Nodes page in SolarWinds, Add a new Node
    4. For the Node name enter victorops.com

     

    Create an External Node for VictorOps.com

    1. From the Manage Nodes page in SolarWinds, Add a new Node
    2. For the Node name enter victorops.com

    The application template can be assigned to any node; however, this step creates a more visually appealing look in your SolarWinds instance.

    Edit the VictorOps Ack-Back Application Template

    1. If you have not already import the VictorOps Ack-Back application template from thwack
    2. Select the VictorOps Ack-Back application template under Manage Application Monitor Templates, click Edit
    3. Note the Polling Frequency
      1. The Polling Frequency is set to 60 seconds. I would recommend leaving this custom setting to avoid ack-back delays.
    4. Expand the Component Ack-Back Get Requests
    5. Change the Credential for Monitoring to the SolarWinds API User

    6. Update the PowerShell script with your VictorOps API Company ID, API Key and SolarWinds primary server name. You can find the API values from your VictorOps portal under Settings > API

                    Near the top of the powershell script you will find two variables, $API_ID and $ApiKey. Enter your API company ID and API key from VictorOps.
             #Victor Ops Company ID
             $API_ID = "Your ID"
             #Victor Ops API Key
             $ApiKey = "Your API Key"
             #Set to your SolarWinds Primary Application Server
             $SolarWindsServer = "Your Primary SolarWinds Server Name"
    7. Click Submit to finish

     

    Assign the VictorOps Ack-Back application template

    1. Assign the VictorOps Ack-Back application template to the victorops.com node created earlier.

     

    Posting Alerts to VictorOps

    There are several nuances to correctly posting alerts to victorops compared to email alerts. Below are some best practices for creating the HTTP post in SolarWinds.

    Trigger and Reset Actions

    Within any alert rule you will need to create a new action under the Trigger and Reset Actions sections.

     

     

    Name of Action: Send to VictorOps

    Action Type: Send a Get or Post Request to a Web Server

    URL: https://alert.victorops.com/integrations/generic/20131114/YourGeneratedURL/YourRoutingKey

    • YourGeneratedURL can be found once you have enabled the REST Endpoint in VictorOps under Settings > Integrations > REST Endpoint
    • YourRoutingKey can be any routing key you have setup in VictorOps. Routing Keys are created when you create schedules in VictorOps. Below is an example of where you can find the Route Key under Settings > Schedules

    Use HTTP POST: Selected

    Body to Post:

    At a minimum your body needs the required fields below. You can add in any custom fields you want. Reference the Examples further below for correct syntax. In general, you follow the below format.

     

    {

    "field_name":"SolarWinds Data",

    "field_name":"SolarWinds Data"

    }

    Required Fields (in order)

     

    Field

    Value

    Notes

    alert_rule

    ${N=Alerting;M=AlertName}

     

    entity_display_name

    Same line as an email subject

     

    entity_id

    ${N=Alerting;M=AlertObjectID}

     

    host_name

    ${NodeName}

     

    ip_address

    ${Node.IP_Address}

     

    message_type

    CRITICAL or RECOVERY

    CRITICAL is Used for the Trigger, RECOVERY for the Reset

    monitor_name

    SolarWinds

     

    monitoring_tool

    SolarWinds

     

    state_message

    Same line as an email subject

     

     

    Screen Shot of a Trigger Action

    Body Post Examples

    Node Is Down

    Trigger Action

    {

    "message_type":"CRITICAL",

    "monitor_name":"SolarWinds",

    "monitoring_tool":"SolarWinds",

    "alert_rule":"${N=Alerting;M=AlertName}",

    "state_message":"${NodeName} is ${Status}",

    "entity_display_name":"${NodeName} is ${Status}",

    "entity_id":"${N=Alerting;M=AlertObjectID}",

    "host_name":"${NodeName}",

    "ip_address":"${Node.IP_Address}"

    }

     

    Reset Action – Note we use RECOVERY for the message_type value

    {

    "message_type":"RECOVERY",

    "monitor_name":"SolarWinds",

    "monitoring_tool":"SolarWinds",

    "alert_rule":"${N=Alerting;M=AlertName}",

    "state_message":"${NodeName} is ${Status}",

    "entity_display_name":"${NodeName} is ${Status}",

    "entity_id":"${N=Alerting;M=AlertObjectID}",

    "host_name":"${NodeName}",

    "ip_address":"${Node.IP_Address}"

    }

     

    Volume Space Alert

    Trigger Action

    {

    "alert_rule":"${N=Alerting;M=AlertName}",

    "entity_display_name":"${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free",

    "entity_id":"${N=Alerting;M=AlertObjectID}",

    "host_name":"${NodeName}",

    "ip_address":"${Node.IP_Address}",

    "message_type":"CRITICAL",

    "monitor_name":"SolarWinds",

    "monitoring_tool":"SolarWinds",

    "state_message":"${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free"

    }

     

    Reset Action

    {

    "alert_rule":"${N=Alerting;M=AlertName}",

    "entity_display_name":"${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free",

    "entity_id":"${N=Alerting;M=AlertObjectID}",

    "host_name":"${NodeName}",

    "ip_address":"${Node.IP_Address}",

    "message_type":"RECOVERY",

    "monitor_name":"SolarWinds",

    "monitoring_tool":"SolarWinds",

    "state_message":"${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free"

    }

     

    Component Based Alert

    Trigger Action

    {

    "message_type":"CRITICAL",

    "monitor_name":"SolarWinds",

    "monitoring_tool":"SolarWinds",

    "alert_rule":"${N=Alerting;M=AlertName}",

    "state_message":"${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",

    "entity_display_name":"${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",

    "entity_id":"${N=Alerting;M=AlertObjectID}",

    "host_name":"${NodeName}",

    "ip_address":"${Node.IP_Address}"

    }

     

    Reset Action

    {

    "message_type":"RECOVERY",

    "monitor_name":"SolarWinds",

    "monitoring_tool":"SolarWinds",

    "alert_rule":"${N=Alerting;M=AlertName}",

    "state_message":"${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",

    "entity_display_name":"${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",

    "entity_id":"${N=Alerting;M=AlertObjectID}",

    "host_name":"${NodeName}",

    "ip_address":"${Node.IP_Address}"

    }

     

    Custom Fields

    Node Based Alerts

    "info":"${N=SwisEntity;M=CustomProperties.Info}",

    "team":"${N=SwisEntity;M=CustomProperties.Team}"

     

    Application and Component Based Alerts

    "info":"${N=SwisEntity;M=Application.Node.CustomProperties.Info}",

    "team":"${N=SwisEntity;M=Application.Node.CustomProperties.Team}"