Slack Integration with alerting.

Question

I have an alert that has two slack integrations, one on escalation level one, one on escalation level two.

Escalation level one, always works, escalation level two always gives an error:action has failed Failed to execute HTTP request

The content of the JSON is identical apart from some change of text:

Level 1:

payload={"text":"${N=SwisEntity;M=Node.DisplayName} has too many active sessions. Current sessions level is ${N=SwisEntity;M=CustomPollerStatusScalar.Rate}", "attachments":[{"text":"Alert: ${N=Alerting;M=AlertName}.${N=Alerting;M=AlertDetailsUrl} : ${N=Alerting;M=AcknowledgeUrl} notify Technical operations ", "title": "${N=SwisEntity;M=Node.DisplayName} Too many Active sessions.  ","title_link": "${N=SwisEntity;M=DetailsUrl}","color":"#FF0000" }]}

Level 2:

payload={"text":"${N=SwisEntity;M=Node.DisplayName} Sessions cleared. Current sessions level is ${N=SwisEntity;M=CustomPollerStatusScalar.Rate}", "attachments":[{"text":"Alert: ${N=Alerting;M=AlertName}.${N=Alerting;M=AlertDetailsUrl} : ${N=Alerting;M=AcknowledgeUrl} notify Technical operations ", "title": "${N=SwisEntity;M=Node.DisplayName} Sessions cleared.  ","title_link": "${N=SwisEntity;M=DetailsUrl}","color":"#FF0000" }]}

When I use the simulate tool, both work.

jm_sysadmin · Answer

I agree it makes no sense. Hopefully when you get the JSON form the post that fails something jumps out.

michael.kent · Answer

What I don't understand is the top one works, the second one doesn't and the only thing different is the escalation level, they are in the same alert.

Some text has changed, none of the formatting has.

jm_sysadmin · Answer

It drives me nuts when the issue doesn't happen in the simulate tool. I would add lines to log the payload actions to a file. My guess is that something doesn't show up right from the variables (or you need to clean up data from a variable by adding an escape character or just replacing it) and the payload formatting is off, which causes it to fail. I'd write the whole JSON blob so you can manually recycle it for your own post and see the exact behavior. Either that or simplify it down to the bare minimum and add content back until it errors or works.

Only thing similar I have seen was some of my custom properties had "invisible" characters from the system that exported them. Still not sure what they were, but now those properties I do a regex replace to only allow letters and numbers, and my script stopped failing at weird times. I am not saying it related, but until you see what it tried to post, we are likely in the dark since the simulate works. The good news is that the major parts must work, its just some little detail it trips up on.