While trying to get events to fire in PagerDuty I ran into inconsistencies. For now, please ignore the verbose content. I inherited these and am working on cleaner alert messages.
Let's start with the event in the event viewer with things fail:
Failed to execute an HTTP request. Method: POST, Url: events.pagerduty.com/.../enqueue, Body: { "payload": { "summary": "MyNode.xx.yyy.net is Down.", "source": "MyNode.xx.yyy.net", "severity": "critical", "component": "Cisco Catalyst xxxx+y", "custom_details": { "Description": "Cisco IOS Software, IOS-xx Software, Catalyst xxxx L3 Switch Software (catxxxx-UNIVERSALK9-M), Version 0x.0x.0x.x RELEASE SOFTWARE (fc4) Technical Support: www.cisco.com/techsupport Copyright (c) 1986-2018 by Cisco Systems, Inc. Compiled Thu", "Application Name": "SolarWinds.Core.Common", "Node": "MyNode.xx.yyy.net", "Node Availability": "Node status is Down, Percent Packet Loss threshold is Warning." } }, "routing_key": "MyKey", "dedup_key": "MyNode.xx.yyy.net_MyNode.xx.yyy.net_Cisco Catalyst xxxx+y", "event_action": "trigger", "client": "SolarWinds Orion", "client_url": "">someserver.yy.yy.com:80/.../NodeDetails.aspx })
Notice the stray ")" at the end? AFAIK this is not part of the JSON payload (See Below). This SEEMS to be causing the failure, but, I don't know with 100% certainty if this ) really IS getting sent to PagerDuty, or if this is simply something added into the logging of the failure?
After much testing, I determined this var: "${N=SwisEntity;M=NodeDescription}" Seems to be involved in the problem. This is the content of the var:
Cisco IOS Software, IOS-xx Software, Catalyst xxxx L3 Switch Software (catxxxx-UNIVERSALK9-M), Version 0x.0x.0x.x RELEASE SOFTWARE (fc4) Technical Support: www.cisco.com/techsupport Copyright (c) 1986-2018 by Cisco Systems, Inc. Compiled Thu
Here's where it gets interesting.... In the JSON payload, I can remove the var ${N=SwisEntity;M=NodeDescription} and replace it with the literal text block (the actual contents of the var) and everything WORKS! Yes, it's the only difference between a trigger action that works and one that doesn't. I thought perhaps the slashes in the url weren't getting escaped properly - nope, that's not it. The only thing I can think of is the parser is getting confused with all the ( )'s in the var during expansion.
Here's the JSON in the trigger action:
{
"payload": {
"summary": "${N=SwisEntity;M=Caption;F=OriginalValue} is ${N=SwisEntity;M=Status;F=Status}.",
"source": "${N=SwisEntity;M=Caption}",
"severity": "critical",
"component": "${N=SwisEntity;M=MachineType}",
"custom_details": {
"Description": "${N=SwisEntity;M=NodeDescription}",
"Application Name": "${N=Generic;M=Application}",
"Node": " ${N=SwisEntity;M=Caption;F=OriginalValue}",
"Node Availability": "${N=SwisEntity;M=StatusDescription}"
}
},
"routing_key": "MyKey",
"dedup_key": "${N=SwisEntity;M=Caption;F=OriginalValue}_${N=SwisEntity;M=Caption}_${N=SwisEntity;M=MachineType}",
"event_action": "trigger",
"client": "SolarWinds Orion",
"client_url": "${N=SwisEntity;M=DetailsUrl}"
}
Again, I know there is a bunch of unnecessary stuff here - I'll deal with that. I can resolve my current issue easy enough, just wanted to get some feedback as if there is a parsing or formatting issue it could bite someone else and maybe there's not an easy workaround.
TIA - Gary
