Automating Incident Creation With Orion

As of Orion Core version 2019.4, SolarWinds Service Desk has native integration with the Orion Platform.

When we launched SolarWindsRegistered Service Desk (SWSD), I couldn’t wait to get my hands on it. I was very excited to see a new solution to handle incident management, asset management, an internal knowledge base, problem management, and an employee self-service portal. There’s so much to this new product to unpack, I needed to figure out where to start. Thankfully, there was already an excellent document introducing everyone to the solution I could read.

For the past three years, I’ve been getting deeper and deeper into leveraging various APIs to do my bidding. This lets me go nuts on my keyboard and automate out as many repeatable functions as possible. No, I’m not breaking up with my mouse. We had a healthy discussion, my mouse and I, and he’s fine with the situation. Really. What was I talking about? Oh yeah, APIs!

One of the things I absolutely love about working with APIs (and scripting languages as well) is there’s no one way to do something. If you can think it, you can probably do it. Most RESTful APIs allow you to work with whatever language you prefer. You can use the curl executable, Perl (tagging Leon here), PowerShell, or nearly anything else. PowerShell is my personal preference, so I’m doing my scripting with it. But more on those details later.

You’ve seen me write and talk about using the SolarWindsRegistered OrionRegistered API to help automate your monitoring infrastructure. I’ve even gotten some of my friends in on the trend. But, the launch of SWSD opened a brand-new API for me to explore. I started where I always do with something new: by reading the manual. SolarWinds Service Desk has extensive documentation about using the API. There’s so much there for me to explore, but I had to limit myself. In trying to pick a place to start, I thought about my past.

SolarWinds has always been in the business of helping IT professionals do their jobs better. Many of us technology professionals, like me, started our careers working on a help desk. Based on everything SWSD offers, I limited myself to the Incidents Management area. Then I just had to think about how I would leverage this type of solution in some of my previous roles.

As a help desk supervisor who went on to be a monitoring engineer, I thought about how great it would be to get tickets automatically created based on an alert. I could talk all day about what qualifies for an alert (I have) and what’s best to include in an alert message (that, too), but the biggest thing to strive towards is some level of tracking. The most common tracking method for alerts has been email notifications. This is the default for most people, and 90% of the time it’s fine. But what about the times when email is the problem? You need another way to get your incidents reported and tracked.

Like scripting languages, the Orion alerting engine allows for multiple ways to handle alert logic—not just for the trigger conditions, but also for the actions when the trigger occurs. One of those mechanisms is to execute a program. On the surface, this may sound boring, but not to me and other keyboard junkies. This is a great way to leverage some scripting and the SWSD API to do the work for us.

First things first, we need to decide how to handle the calls to the API. The examples provided in the API documentation use the curl program to do the work, but I’m not in love with the insanely long command lines required to get it to work. But since this is a RESTful API, I should be able to use my preferred scripting language, PowerShell. (I told you I’d get back to it, didn’t I?)

Let’s assemble what you need to get started. First you need your authentication. If you’re an administrator in SWSD, you can go to Setup, Users & Access, and then select yourself (or a service account you want to use). Inside the profile, you’ll find the JSON web token.

pastedImage_2.png

This is how you authenticate with the SWSD API. The web token is a single line of text. In the web display, it’s been wrapped for visual convenience. Copy that line of text and stash it somewhere safe. This is basically the API version of “you.” Protect it as you would any other credentials. In a production system, I’d have it set up to use the service account for my Orion installation.

API Test

For the API call, we need to send over some header information. Specifically, we need to send over the authorization, the version of the API we’ll be using, and the content type we’ll be sending. I found these details in the API documentation for Incidents. To start things off, I did a quick test to see if I could enumerate all the existing incidents.

I’m trying to get more comfortable with JSON, so I’m using it instead of XML. In PowerShell, the HTTP header construction looks like this:

$JsonWebToken = "Your token goes here. You don't get to see mine."

$Headers = @{ "X-Samanage-Authorization" = "Bearer $JsonWebToken";

              "Accept"                   = "application/vnd.samanage.v2.1+json"

              "Content-Type"             = "application/json" }

Basically, we’re saying (in order): this is me (auth), I’d like to use this version of the API with JSON (accept), and I’m sending over JSON as the request itself (content-type).

This block of headers is your pass to speak with the API. I’m testing this from the United States, so I’ll use the base URI via https://api.samanage.com/. There’s a separate one specifically for EU people (https://apieu.samanage.com). If you are in the EU, that’s the one you should be using.

To list out the incidents, we make an HTTP GET call to the “incidents” URI as specified in the documentation. I saved this as a variable so I wouldn’t have copy/paste failures later.

$URI = "https://api.samanage.com/incidents.json"

Then to get the list of all incidents, I can just invoke the REST method.

Invoke-RestMethod -Method Get -Headers $Headers -Uri $URI

pastedImage_3.png

Excellent! I can talk to the API and get some information back. This means I’m authenticating correctly and getting the list of incidents back. Time to move on.

Creating a Test Incident

To create an incident, I only technically need three fields: name (of the incident), the requester, and the title. I’ve seen this called the payload, the body, or the contents. To stay on the same page with the PowerShell parameters, I’ll refer to it as the body. Using it, I built a very small JSON document to see if this would work using the script I’ve started developing. The beauty of it is I can repeatedly use the header I already built. I’ve put the JSON in a string format surrounded by @” and “@. In PowerShell this is called a here-string and there are many things you can do with it.

$TestBody = @"

{

"incident": {

   "name":        "Testing Incident - Safe to Close with no notes",

   "priority":    "Critical",

   "requester":   { "email" : "kevin.sparenberg@kmsigma.com" }

}

}

"@

Invoke-RestMethod -Method Post -Headers $Headers -Uri $URI -Body $TestBody

When I run it, I get back all kinds of information about the incident I just created.

pastedImage_4.png

But to be really, doubly sure, we should check the web console.

pastedImage_6.png

There it is. I can create an incident with my script.

So, let’s build this into an actual alert script to trigger.

Side note: When I “resolved” this ticket, I got an email asking if I was happy with my support. Just one more great feature of an incident management solution.

pastedImage_8.png

Building the new SolarWinds Service Desk Script

For my alert, I’m going with a scenario where email is probably not the best alert avenue: your email server is having a problem. This is a classic downstream failure. We could create an email alert, but since the email server is the source, the technician would never get the message.

pastedImage_9.png

The above logic looks for only nodes with names containing “EXMBX” (Exchange Mailbox servers) and when the status is not Up (like Down, Critical, or Warning).

Now that we have the alert trigger, we need to create the action of running a script.

For a script to be executed by the Orion alerting engine, it should “live” on the Orion server. Personally, I put them all in a “Scripts” folder in the root of the C: drive. Therefore, the full path to my script is “C:\Scripts\New-SwsdIncident.ps1”

I also need to tweak the script slightly to allow for command line parameters (how I send the node and alert details). If I don’t do this, then the exact same payload will be sent every time this alert triggers. For this example, I’m just sticking with four parameters I want to pass. If you want more, feel free to tweak them as you see fit.

Within a PowerShell file, you access command line parameters via the $args variable, with the first argument being $args[0], the next being $args[1], and so on. Using those parameters, I know I want the name of the alert, the details on the alert, the IP of the node, and the name of the node. Here’s what my script looks like:

pastedImage_10.png

You can see I added a few more fields to my JSON body so a case like this could be routed easier. What did I forget? Whoops, this should have said this was a test incident. Not quite ready for production, but let’s move on.

When we build the alert, we set one of the trigger actions as execution of an external program and give it an easily recognizable name.

pastedImage_11.png

The full command line I put here is:

C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -File "C:\Scripts\New-SwsdIncident.ps1" "${N=SwisEntity;M=StatusDescription}" "${N=SwisEntity;M=Caption}" "${N=SwisEntity;M=IP_Address}" "${N=Alerting;M=AlertName}"

This is the path and executable for PowerShell, the script file we want to execute, and the parameters (order is important) we want to pass to the script. I’ve also surrounded the parameters with double quotes because they *may* contain spaces. In this case, better safe than sorry.

Then I just need to sit back and wait for an alert matching my description trigger. There’s one now!

pastedImage_13.png

Just like every alert I write, I’ve already found ways to improve it. Yes, I know this is a very rudimentary example, but it’s a great introduction to the integrations possible. I’ll need to tweak this alert a little bit before I’d consider it ready for prime time, but it’s been a great learning experience. I hope you learned a little bit along with me.

So, I ask you all: where should I go next?

Thwack - Symbolize TM, R, and C