The Incomplete Guide to Integrating SolarWinds Orion and Slack

I started this project yesterday (using a different Thwack post) but this guide is just so darn complete.

Thank you for this!!!

One thing I'll add to this is that, basically any Alert Variable can be used in the Slack message (along with the Slack emoticons if you're so inclined).  I'm not sure if that was listed in the guide at the end since I kind of glossed over that portion (not intentionally, my day got busy - quick).  Below, I've added in a few examples of my messaging that have the IP address of the node along with the Total Down Time and a Slack emoticon (because I couldn't help myself).  One thing to note is that I change the "Alert Description" to be a simplified version of what the alert is so that I can use that when Slack alerts me on my phone, the message title is easy to read (ex. SolarWinds - NODE DOWN   -   That's the name of the "bot" and changes with each alert depending on the node's problem)

I've been having way too much fun with Slack alerting through Solarwinds.  It's like alerting crack.

Examples (scrubbed of identifiers):

This is for a NODE DOWN Alert:

payload={"username": "SolarWinds - ${N=Alerting;M=AlertDescription}","text": "Please help me - My name is *${N=SwisEntity;M=Caption}* and right now I'm *${N=SwisEntity;M=Status}* and scared. \n\n I live at *${N=SwisEntity;M=IP_Address}* and I've been broken for *${N=Alerting;M=DownTime}* minutes. \n\nPlease help!  :cry:"}

This is for a NODE REBOOT Alert:

payload={"username": "SolarWinds - ${N=Alerting;M=AlertDescription}","text": "Please help me - My name is *${N=SwisEntity;M=Caption}* and I think something may be broken. Everything suddenly went dark and then I woke up.  \n\n I remember being at *${N=SwisEntity;M=IP_Address}* when it happened. But it's a little fuzzy after that.  \n\n I need you to find out what happened to me. \n\n Please help!  :face_with_head_bandage:"}

These are awesome, and you (and your company) get props for having a sense of humor as well as a cool Slack integration!

The last chapter of of the manual talks about EXACTLY this - putting more information into an alert helps everyone (and costs you almost NOTHING except a few extra minutes to develop the alert message.).

So I wholeheartedly support this. Well done!!

First, thanks Leon for the write up and mentioning it in the SWUG in Columbus. I am loving this so far, and brandon.blaze inspired me to post my tweak.

So I loved the use of Slack, but sending all the alerts to one channel got a little noisy. I also wanted to get more granular in how Slack notifies me, which you can set by the channel. With emailed alerts we had custom properties already set up both for email addresses to send to, and the names of the groups that match our Support Groups both organizationally and in our ticketing system. Each node has a custom property that has one or more team.

I wanted to send to a channel that matched the team name in that custom property, but couldn’t, those names didn’t match the no caps, no space, only 21 character rules. I could set that in the purpose of the channel, so I started by building out channels with names that were close enough to the ones in use, and setting the purpose to match the values in our custom properties. I did it by hand, but you could build a function to build it as you need it, there is an API for creating a channel, I just didn’t want to tackle that now.

I went to figure out the API, and they definitely let you do this, but I need a different access token.

Go here for the Token: https://api.slack.com/web

Here for the list channel API: https://api.slack.com/methods/channels.list

My additional powershell code is below, and you can see

function ChannelToPostTo ($OrionProperty)

{

       $token = '<Your Token that you got from https://api.slack.com/web>' 

       $PostListRequest = @{ token = $Token }

       $channel = (Invoke-RestMethod -Uri https://slack.com/api/channels.list -Body $PostListRequest).channels | Select Name, Purpose | Where-Object { $OrionProperty -contains $_.purpose.value }

       return '#'+$channel.name

}

foreach ($group in $SupportGroup.Split(","))

{

       $GroupChannel = ChannelToPostTo($group.Trim())

       If ($GroupChannel -eq "#") {$GroupChannel = "#general"}

       $slackJSON = @{ }

       $slackJSON.channel = $GroupChannel

       #< all the code from Leon or what you built to send the message>

       Invoke-WebRequest @webReq

}

First I add a parameter to  populate to $SupportGroup from the alert variable just like the other in Leon's example.

The function does a look up of all the channels, selects the things I cared about, and pulls the channel name that equals the one that I set the purpose to. I return the name with the # sign. Since I know that I have occasionally have multiple groups in the property, I send the notice to each channel with the for each loop, and set the JSON channel with the variable I built. The rest is just like the example from Leon or whatever you do.

If I don’t have a support team for whatever reason, I send it to general (an advantage over email which goes nowhere). You could replace that with a function that creates a room, but we won’t change that much, and I’d rather not make a bunch of rooms by mistake or for something that almost never happens.

Let me know if you see room for improvement or anything that you have thought to do

jm_sysadmin stop slacking off and get back to work. BTW thanks for sharing.

Teamwork means I can slack off while you work right?

For the messages that get sent to #GENERAL that don't have categorization and you don't want to create a ton of groups, it may be possible to send a Slack Direct Message to yourself or the individual responsible for acknowledging the alert.  I haven't played around with the idea of DMs yet through alerting, but you have inspired me and I will test and post the results.

This Slack integration alerting is way more fun than should be allowed.  (I have never felt as nerdy as when I was typing out that previous sentence  )

‌Good idea, general is shown to all so it may not be the best place, I think either will add a miscellaneous channel or send it to our operations team that gets staffed 24/7. I likely won't know a specific person to send to

i like to think my post was the inspiration for this good work chaps

It certainly was (for me at least).  I use the format from your post for the Slack alerts as I'm not one to go powershelling my way through anything (much to my own detriment).