This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Alerting using #Slack

Hi,

Thought i'd put this up here, as i've found it massively useful - and it has completely changed out alerting strategy - for the better. Those of you that don't know, Slack is a messaging / collaboration tool - allowing you to create channels for discussions on different subjects across your team. All IT teams in our company use it across 3 sites, and it is a great tool. One of the best things about it, is that you can send information to slack using the API.

This enables us to have channels dedicated to Alerts triggered from Solarwinds. People that need to know about things subscribe to the relevant channels, then others that might want to know, but don't need to be alerted can also subscribe but turn off notifications. The up shot is, that everyone knows where to look first in the event of an issue - especially for IT users that don't use solar winds directly (i.e. devs that might be deploying code having issues etc) Slack has a windows app with desktop notifications, iPhone / android apps with push notifications - so alerts can be actioned by the right people straight away. It also triggers conversation on the problem, which is great. here is an example of a couple of our simple alerts:

slack1.PNG

slack2.PNG

As you can see, you can pass information in the same way you can emails, so any useful information to make the alert more descriptive the better. Reboot alert allows users to just click the link to acknowledge they rebooted it (and notify the channel that they did the reboot)

The same goes for reset actions

slack3.PNG

You get the idea.

Configuring these alerts is just simple JSON - Slack is has a free tier, so there is no cost in using it - once you have signed up you can then manage your integrations on the preferences page.

For Slack:

Create a channel in Slack for your alerts

Create a new Integration > choose Incoming webhook

You will be given a Webhook URL - make a note of it

Choose which channel the alerts will go to

For Solarwinds:

For any currently configured alert, you can just add a new alert action Send a GET or POST to a web server

In the Configure action change the radio button to use HTTP POST

In the URL use the URL you were given in the Slack incoming webhook configuration

In the body to post, this is what will be posted and should be in JSON format. Solarwinds for some reason doesnt like the 'proper' format, so example, you can't have line breaks or spaces to make the code easier to read, you need to have it as 1 line. This is a pain for editing and working out what you want to post, so i recommend using sublime or some other text editor to write it out, then copy and paste it into the alert. So to get the Reboot alert:

payload={"text":"${N=SwisEntity;M=Caption} Has Rebooted", "attachments":[{"text":"Alert: ${N=Alerting;M=AlertName}.${N=Alerting;M=AlertDetailsUrl} If this was you, please acknowledge here: ${N=Alerting;M=AcknowledgeUrl} ", "title": "${N=SwisEntity;M=Caption} has rebooted.  ","title_link": "${N=SwisEntity;M=DetailsUrl}","color":"#FEE529" }]}

There are different parts to the alert, so you have the title, then the text, then the attachments text - there might be a better way to lay this out, but i couldnt get it to work. The variables can be anything that you can put in an email or any other alert. the color setting is just the color in the slack notification.

So that's pretty much it - its a simple way of getting people to chat and discuss alerts without distracting emails to people that don't care, all level 1 alerts we have now come into slack channels, separated by team or service. you don't end up with an inbox full of useless alerts, notifications are pushed to mobile too - so everyone is up to speed regardless of where they are.

  • We have been using this for about 6 months, and the customization available for the slack posts is fantastic!

    I can attach some examples of our various post types if anyone is interested.

  • Yes,could you share some examples?  Thanks!

  • We have our own ticketing/device portal that we keep all of our device information in, so I have links in each applicable alert to their device information as well.  We use a custom property with a device ID that is unique in our portal.

    Application Alert

    slack Application Alert.PNG

    {

        "attachments": [

            {

                "fallback": "${N=Alerting;M=AlertName}",

      "pretext": "${N=Alerting;M=AlertName}",

                "title": "Application: <http://<SW URL>/Orion/APM/ApplicationDetails.aspx?NetObject=AA:${N=SwisEntity;M=ApplicationID} |${N=SwisEntity;M=ApplicationAlert.ApplicationName} >",

                "fields": [

      {

      "title": "Components with Problems",

      "value": "${N=SwisEntity;M=ApplicationAlert.ComponentsWithProblemsFormatted}  ",

      "short": false

      },

      {

      "title": "Time of Status Change",

      "value": "${N=SwisEntity;M=CurrentStatus.ObservationTimestamp;F=Time}",

      "short": false

      },

      {

      "title": "Customer Name",

      "value": "<https://<SW URL>/rp/customers/show/${N=SwisEntity;M=Node.CustomProperties.CustomerName} |${N=SwisEntity;M=Node.CustomProperties.CustomerName}>",

      "short": true

      },

      {

      "title": "Asset Tag",

      "value": "<https://<SW URL>/rp/devices/show/${N=SwisEntity;M=Node.NodeID}${N=SwisEntity;M=Node.CustomProperties.AssetTag} |${N=SwisEntity;M=Node.CustomProperties.AssetTag}>",

      "short": true

      },

      {

      "title": "Alert Details",

      "value": "<${N=Alerting;M=AlertDetailsUrl}|Alert Link>",

      "short": true

      },

      {

      "title": "Acknowledge",

      "value": "<${N=Alerting;M=AcknowledgeUrl}|Ack Link>",

      "short": true

      }

                ],

                "color": "danger"

            }

        ]

    }

    Node Down Alert

    slack Node Down Alert.PNG

    {

        "attachments": [

            {

                "fallback": "${N=Alerting;M=AlertName}",

      "pretext": "${N=Alerting;M=AlertName}",

                "title": "Node Down Alert on <http://<SW URL>/Orion/NetPerfMon/NodeDetails.aspx?netobject=N:${N=SwisEntity;M=NodeID}&view=NodeDetails|${N=SwisEntity;M=Caption}>",

      "fields": [

      {

      "title": "Customer Name",

      "value": "<https://<OUR DEVICE LINK>/${N=SwisEntity;M=CustomProperties.CustomerID}|${N=SwisEntity;M=CustomProperties.CustomerName}>",

      "short": true

      },

      {

      "title": "Asset Tag",

      "value": "<https://<OUR DEVICE LINK>/rp/devices/show/${N=SwisEntity;M=CustomProperties.DeviceID}|${N=SwisEntity;M=CustomProperties.AssetTag}>",

      "short": true

      },

      {

      "title": "Alert Details",

      "value": "<${N=Alerting;M=AlertDetailsUrl}|Alert Link>",

      "short": true

      },

      {

      "title": "Acknowledge",

      "value": "<${N=Alerting;M=AcknowledgeUrl}|Ack Link>",

      "short": true

      }

                ],

                "color": "danger"

            }

        ]

    }

    I have these for all of our different alerts, these are the most generic that could be used most often I'd think.

    Hope this helps!

    Jack

  • I would advise everyone to tune into SolarWinds lab #40 (which is the February episode) when we go over EXACTLY this information!

    This is awesome stuff. Keep it coming.

    Hey patrick.hubbard‌ do you see these guys? It's like we're all on the same team or something!

  • I just need to work out how to get Hubot to resolve the issues for me. The AI required to clean up disk space can't be too hard right ?

  • Is there a simple way to escape backslashes (e.g. in Windows volume names) in variables?

    A simplified alert I'd like to post:

    payload={"text": "Warning: <${N=SwisEntity;M=DetailsUrl}|${VolumeDescription}> on <${N=SwisEntity;M=DetailsUrl}|${NodeName}>: ${VolumePercentUsed} used"}

    The problem: when a variable contains backslashes (like a Windows volume name, e.g. "C:\"), this won't get posted: backslashes need to be "escaped" by another backslash in front of it, e.g. "C:\\".

    Is there a way to make this work for backslashes in the variables?

  • I'm pretty sure that i never had this issue before NPM 12 - i.e. was able to send a disk utilisation with the c:\ in it - but now can't!

    Work around is:

    ${SQL: SELECT replace(Caption, '\', '\\') FROM Volumes WHERE VolumeID = ${VolumeID}} on ${NodeName} is ${VolumePercentUsed}.

    I have that in the Message Displayed when triggered box in the alert, then call that in the alert when it fires:

    {

        "attachments": [

            {

                "fallback": " ${N=Alerting;M=AlertMessage}",

                "pretext": "New Alert Raised",

                "title": "${N=Alerting;M=AlertName}",

                "title_link": "${N=Alerting;M=AlertDetailsUrl}",

                "text": "  ${N=Alerting;M=AlertMessage}, ${N=Alerting;M=AlertDescription}  ",

                "color": "#FF5757"

            }

        ]

    }

    11111.png

  • FormerMember
    0 FormerMember

    This post was about #SLACK, but has anyone got it work with Hipchat?

    In Hipchat, clicking "integrations" and "creating your own integrations" I'm given an app url:

    https://company.hipchat.com/v2/room/2896360/notification?auth_token=h6avZv65RKD4eEAKwIsIvbLsZxr8AJK0hkI4

    Using this in http POST in solarwinds results in "Failed to execute HTTP request"


  • Perfect, thank you - it works!