cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 11

Alerting using #Slack

Hi,

Thought i'd put this up here, as i've found it massively useful - and it has completely changed out alerting strategy - for the better. Those of you that don't know, Slack is a messaging / collaboration tool - allowing you to create channels for discussions on different subjects across your team. All IT teams in our company use it across 3 sites, and it is a great tool. One of the best things about it, is that you can send information to slack using the API.

This enables us to have channels dedicated to Alerts triggered from Solarwinds. People that need to know about things subscribe to the relevant channels, then others that might want to know, but don't need to be alerted can also subscribe but turn off notifications. The up shot is, that everyone knows where to look first in the event of an issue - especially for IT users that don't use solar winds directly (i.e. devs that might be deploying code having issues etc) Slack has a windows app with desktop notifications, iPhone / android apps with push notifications - so alerts can be actioned by the right people straight away. It also triggers conversation on the problem, which is great. here is an example of a couple of our simple alerts:

slack1.PNG

slack2.PNG

As you can see, you can pass information in the same way you can emails, so any useful information to make the alert more descriptive the better. Reboot alert allows users to just click the link to acknowledge they rebooted it (and notify the channel that they did the reboot)

The same goes for reset actions

slack3.PNG

You get the idea.

Configuring these alerts is just simple JSON - Slack is has a free tier, so there is no cost in using it - once you have signed up you can then manage your integrations on the preferences page.

For Slack:

Create a channel in Slack for your alerts

Create a new Integration > choose Incoming webhook

You will be given a Webhook URL - make a note of it

Choose which channel the alerts will go to

For Solarwinds:

For any currently configured alert, you can just add a new alert action Send a GET or POST to a web server

In the Configure action change the radio button to use HTTP POST

In the URL use the URL you were given in the Slack incoming webhook configuration

In the body to post, this is what will be posted and should be in JSON format. Solarwinds for some reason doesnt like the 'proper' format, so example, you can't have line breaks or spaces to make the code easier to read, you need to have it as 1 line. This is a pain for editing and working out what you want to post, so i recommend using sublime or some other text editor to write it out, then copy and paste it into the alert. So to get the Reboot alert:

payload={"text":"${N=SwisEntity;M=Caption} Has Rebooted", "attachments":[{"text":"Alert: ${N=Alerting;M=AlertName}.${N=Alerting;M=AlertDetailsUrl} If this was you, please acknowledge here: ${N=Alerting;M=AcknowledgeUrl} ", "title": "${N=SwisEntity;M=Caption} has rebooted.  ","title_link": "${N=SwisEntity;M=DetailsUrl}","color":"#FEE529" }]}

There are different parts to the alert, so you have the title, then the text, then the attachments text - there might be a better way to lay this out, but i couldnt get it to work. The variables can be anything that you can put in an email or any other alert. the color setting is just the color in the slack notification.

So that's pretty much it - its a simple way of getting people to chat and discuss alerts without distracting emails to people that don't care, all level 1 alerts we have now come into slack channels, separated by team or service. you don't end up with an inbox full of useless alerts, notifications are pushed to mobile too - so everyone is up to speed regardless of where they are.

Tags (1)
25 Replies
Level 7

Has anybody done this with a Telegram bot? I see a few people asking about how to do it, and no answers.  Telegram has an API for channel bots, and I am able to send a message to the bot if I put the channel ID and text in the URL itself.  Documentation says it supports HTTP/S POST, but I can't really find any good examples for what to put in the body.

0 Kudos

I did finally find an answer in another thread.

method to call external api or other real time integration?

0 Kudos
Level 8

Just thought I post it here as well - I had a bit of a different approach for this - see here: WebHelpdesk - Slack Alerts - Scripts

Thought it is good to push out options for those who are interested...

Regards

0 Kudos
Level 9

Great post, very helpful.   For some reason the AlertDetailsURL and AlertAcknowledgeURL are not expanding.  Everything else expands just fine.  Ideas?

Ok, so possibly expanding.  Doesn't expand when doing the TEST function in AlertManager.   When the action actually triggers it does appear in Slack with the appropriate URLs.

0 Kudos

I've had no luck having real alerts properly format the "Alert Details" and "Ack Link" I always get the variable string showing as text.

0 Kudos
Level 8

This post was about #SLACK, but has anyone got it work with Hipchat?

In Hipchat, clicking "integrations" and "creating your own integrations" I'm given an app url:

https://company.hipchat.com/v2/room/2896360/notification?auth_token=h6avZv65RKD4eEAKwIsIvbLsZxr8AJK0...

Using this in http POST in solarwinds results in "Failed to execute HTTP request"


0 Kudos

I was able to get it working by executing a vbscript with the post information from the integrations page in Hipchat. It doesn't like spaces in the alert name title so it replaces the spaces with a hyphen.  Hope it helps.

Alert Action:

Execute an external VB Script

VB Script Interpreter: CScript

copy the script local to the poller to add to the script path.

Example:

c:\scripts\Orion_Alert.vbs ${N=SwisEntity;M=Caption} ${N=SwisEntity;M=IP_Address} ${N=Alerting;M=AlertName}

-----------------------------------------------------------

Here is an example of the vbscript saved as Orion_Alert.vbs:

-----------------------------------------------------------

' Input your sUrl here

sUrl = "https://yourHipChat-sUrl-here.com/v2/room/?/notification?auth_token=???????your-token??????"

sArgName0=WScript.Arguments.Item(0) ' CAPTION

sArgName1=WScript.Arguments.Item(1) ' IP ADDRESS

sArgName2=Replace(WScript.Arguments.Item(2), " ", "-")  'Alert Name with spaces changed to hypen

sReqColor="""color"":""red"""

sReqMessage="""message"":""" + "[" + sArgName2 + "]" + " - " + "Device:" + sArgName0 + "  IP Address:" + sArgName1  + """"

sReqNotify="""notify"":true"

sReqMessage_format="""message_format"":""text"""

sRequest="{" + sReqColor + "," + sReqMessage + "," + sReqNotify + "," + sReqMessage_format + "}"

HTTPPost sUrl, sRequest

Function HTTPPost(sUrl, sRequest)

set oHTTP = CreateObject("Microsoft.XMLHTTP")

  oHTTP.open "POST", sUrl, false

  oHTTP.setRequestHeader "Content-Type", "application/json"

  oHTTP.setRequestHeader "Content-Length", Len(sRequest)

  oHTTP.send sRequest

  HTTPPost = oHTTP.responseText

End Function

cschomburg​  - perhaps we should start a new thread, but even when replacing the spaces with a dash (I also tried an underscore and an asterisk), the VBscript example you provided to send to Hipchat still drops everything after the first space in the ${N=Alerting;M=AlertName} variable   Using your exact script (with my url of course), here is what I get in the HipChat room (server names and IP addresses have been changed to protect the innocent):

SolarWinds·8:42 AM

[Alert] - Device:SERVER.example.com  IP Address:10.10.10.10

If I understand how this works, this notification *should* read:

SolarWinds·8:42 AM

[Alert-me-when-an-application-goes-down] - Device:SERVER.example.com  IP Address:10.10.10.10

Is my assumption correct?  Out of curiosity, is this method working for you in NPM 12?

Thanks in advance!

0 Kudos

@cschomburg
Thank you, but running script locally or orion server returns "vbscript out of range".

Not sure what to replace ' CAPTION with and if I should start after the '

For ' IP ADDRESS should that be the Orion server IP?

Or are these variables?

Changed alert name with spaces changed to hyphen but again not sure where to put it?

"Host-memory-utilization-above-90%", "-")

0 Kudos

be sure to remove those weird characters this forum added like those bubble o's above.   I also took out those 'comment lines.  Load the vbs in the free Mircosoft Visual Studio and it color codes it nicely.  Then run the cscript.exe command on the vbs on the solarwinds server in the directory you created and feed it variables like 1 2 3 and you'll see that appear in your hipchat room.  ex: cscript.exe c:\scripts\myscript.vbs 1 2 3

Cleaned up code below

sUrl = "https://hipchat.prod.xxxxx.com/v2/room/1045/notification?auth_token=xxxxx"

sArgName0=WScript.Arguments.Item(0)

sArgName1=WScript.Arguments.Item(1)

sArgName2=Replace(WScript.Arguments.Item(2), " ", "-")

sReqColor="""color"":""red"""

sReqMessage="""message"":""" + "[" + sArgName2 + "]" + " - " + "Device:" + sArgName0 + " IP Address:" + sArgName1  + """"

sReqNotify="""notify"":true"

sReqMessage_format="""message_format"":""text"""

sRequest="{" + sReqColor + "," + sReqMessage + "," + sReqNotify + "," + sReqMessage_format + "}"

HTTPPost sUrl, sRequest

Function HTTPPost(sUrl, sRequest)

set HTTP = CreateObject("Microsoft.XMLHTTP")

  HTTP.open "POST", sUrl, false

  HTTP.setRequestHeader "Content-Type", "application/json"

  HTTP.setRequestHeader "Content-Length", Len(sRequest)

  HTTP.send sRequest

  HTTPPost = HTTP.responseText

End Function

0 Kudos

also put -'s for EVERYTHING no space in anything, the script doesn't seem to replace the spaces with -'s as it should.  The alert name, the Message Displayed when alert triggered, even the node name.  That's the only way I've gotten it to work. If someone has a cleaner fix please post.  Mine now shows like this in HipChat:

Solarwinds·5:47 PM

[AWS-Region-Unreachable] - Device:AWS-Ireland-Region  IP Address:x.x.x.x

0 Kudos

Finally fixed the little nagging issue of having to use dash's in everything.  When you put in the location of the vbscript with the 3 variables after, just put each variable inside double quotes "$variable1" etc and everything works perfectly thanks cschomburg for starting this

0 Kudos
Level 13

Is there a simple way to escape backslashes (e.g. in Windows volume names) in variables?

A simplified alert I'd like to post:

payload={"text": "Warning: <${N=SwisEntity;M=DetailsUrl}|${VolumeDescription}> on <${N=SwisEntity;M=DetailsUrl}|${NodeName}>: ${VolumePercentUsed} used"}

The problem: when a variable contains backslashes (like a Windows volume name, e.g. "C:\"), this won't get posted: backslashes need to be "escaped" by another backslash in front of it, e.g. "C:\\".

Is there a way to make this work for backslashes in the variables?

0 Kudos

I'm pretty sure that i never had this issue before NPM 12 - i.e. was able to send a disk utilisation with the c:\ in it - but now can't!

Work around is:

${SQL: SELECT replace(Caption, '\', '\\') FROM Volumes WHERE VolumeID = ${VolumeID}} on ${NodeName} is ${VolumePercentUsed}.

I have that in the Message Displayed when triggered box in the alert, then call that in the alert when it fires:

{

    "attachments": [

        {

            "fallback": " ${N=Alerting;M=AlertMessage}",

            "pretext": "New Alert Raised",

            "title": "${N=Alerting;M=AlertName}",

            "title_link": "${N=Alerting;M=AlertDetailsUrl}",

            "text": "  ${N=Alerting;M=AlertMessage}, ${N=Alerting;M=AlertDescription}  ",

            "color": "#FF5757"

        }

    ]

}

11111.png

Perfect, thank you - it works!

0 Kudos
Level 17

I would advise everyone to tune into SolarWinds lab #40 (which is the February episode) when we go over EXACTLY this information!

This is awesome stuff. Keep it coming.

Hey patrick.hubbard‌ do you see these guys? It's like we're all on the same team or something!

Leon Adato | Head Geek
------
"Measure what is measurable,
and make measurable what is not so." - Gallileo

I just need to work out how to get Hubot to resolve the issues for me. The AI required to clean up disk space can't be too hard right ?

Level 9

We have been using this for about 6 months, and the customization available for the slack posts is fantastic!

I can attach some examples of our various post types if anyone is interested.

Interested

Thanks,

Christian

0 Kudos