cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 8

SAM PagerDuty integration issues with volume names

We're using PagerDuty with SAM to handle on-call rosters & escalations.

We've hit an issue with disk alerts where PagerDuty is choking on the "\" character that is part of the volume name we want to include in alerts (eg. "S:\ Label: 86b4c87d").

Basically, when we POST it to PagerDuty it returns with the following error;

Encountered error in GET action. - The remote server returned an error: (400) Bad Request

I looked at the available macros for Volumes, and it seems like ${Caption}, ${VolumeDescription}, and ${FullName} all include the "\" by default.

There are a couple of ways of tackling this that I can think of. One is to create a separate alert for each volume and just hard code the drive letter in the alert text, rather than having SAM insert it via a Macro. It would be better if I can remove the "\" char from one of the available macro fields. Does anyone know if that is possible? Are they customizable?

Has anyone else hit this issue and have a different way of dealing with it? I can't believe I'm the first person to try sending disk alerts to PagerDuty.

0 Kudos
10 Replies
Level 7

I realize this is an old thread but I ran into the same limitation with backslashes in the PagerDuty API. One workaround is to use a REPLACE function in a SQL alert variable: SELECT REPLACE(Caption, '\', '\\') FROM Volumes.

The "description" field in my JSON POST text ended up like this:

"description": "Critical Alert: ${SQL: SELECT REPLACE(Caption, '\', '\\') FROM Volumes WHERE VolumeID = ${VolumeID}} is ${VolumePercentUsed} full - ${VolumeSpaceAvailable} free / ${VolumeSize} total."

Product Manager
Product Manager

This sounds like a newly introduced bug in PagerDuty as there are many in the community who have used this service for years without issue. Is it possible this only occurs when the "\" character exists in the subject line of the email, or is this easily reproducible when the volume name/description is only in the message body text? If you are not able to get PagerDuty to resolve this issue you may find Alert Central provides many of the same capabilities as PagerDuty without the cost and doesn't exhibit this behavior.

0 Kudos

Possibly a new bug - I only just noticed it, so I don't know how long it's been around for.

It's the text in the description field. I don't think the pagerduty api has subject or body fields, just description (see below).

If I hard-code the text in the JSON I send over, "S: Label: 86b4c87d" works fine, but "S:\ Label: 86b4c87d" does not, hence my assumption that the "\" char is screwing things up.

As a workaround, I just split our alerts out into one for each drive letter and just put the drive letter directly into the description field text rather than trying to use a Macro to insert it.

Here's the JSON that gets posted;

{

  "service_key": "blahblahblah",

  "event_type": "trigger",

  "description": "WARNING - ${Node.Caption} - ${VolumeDescription} is ${VolumePercentUsed} full",

  "client": "SolarWinds",

  "incident_key": "${AlertID}-${ObjectID}",

  "details":

  {

    "alert_name": "${AlertName}",

    "node_name": "${NodeName}",

    "alert_id": "${AlertID}",

    "node_id": "${NodeID}",

    "severity": "${Severity}",

    "object_id": "${ObjectID}"

  }

}

0 Kudos

So I heard back from PagerDuty;

"Thank you for bringing this to our attention. Unfortunately, it looks like, due to the nature of Windows-path names, SolarWinds is sending out invalid JSON (verified here:http://jsonlint.com/), which contains unescaped "\" characters, to our integration API. "

So I guess what we have here is a case of Solarwinds and PagerDuty standing and pointing fingers at each other?

0 Kudos
Product Manager
Product Manager

How are you sending this information to PagerDuty? Most customers send email notifications as their primary method of forwarding alerts from Orion to PagerDuty.

0 Kudos

We're using their documented SolarWinds integration method, which involves POSTing some JSON to their site;

https://www.pagerduty.com/docs/guides/solarwinds-npm-integration-guide/

This was something they introduced earlier this year.

0 Kudos
Product Manager
Product Manager

The variable not being properly escaped is an issue in the JSON post method PagerDuty provided as the downloadable trigger action for Orion. The variable is populating properly, but the code they have provided to their customers to integrate with the PagerDuty service does not handle reserved characters like "\" properly.

0 Kudos

Yes, we've already established what's causing the issue. Question is, how to fix it?

What I'm hearing so far from both companies is that it's the other sides fault. Pagerduty are providing a JSON api, Solarwinds is putting characters into the macros that are outside of the acceptable list of JSON characters, without any way of filtering or escaping the text. Seems we're at an impasse.

0 Kudos
Product Manager
Product Manager

The macros available via Orion are either string or integer values which can contain characters that are illegal in XML. Below is a sampling of those characters. This is because these macros have been designed to be used primarily with email notifications, text files, event logs, syslog, traps, etc. not XML. XML requires any illegal character to be escaped inline where the illegal character exists within the string. For example, "Server & Application Monitor" becomes " Server &amp Application Monitor" in XML. If you rely upon PagerDuty for notification I recommend instead of using this restful API that you use Email as the primary method of integration. This is especially true given the dynamic nature of macros in the Orion platform that are based on information polled from systems or user entered information which cannot be guaranteed to be escaped properly. Another option probably worth considering is Alert Central, which provides similar functionality to PagerDuty, is free, integrates directly with Orion and virtually any other management system, but does not suffer from this limitation.

( ' ) - apostrophe

( " ) - quote

( : ) - colon

( / ) - forward slash

( \ ) - back slash

( | ) - Pipe

( . ) - full stop (or period)

( & ) - ampersand

( < ) - less than

( > ) - greater than

0 Kudos

I've been talking with PagerDuty support as well about this, and they've committed to reaching out to SolarWinds to see if the two companies can agree on an approach sanctioned by both sides.

0 Kudos