Send POST > failed to execute http request

I'm currently migrating all of my POST request actions from WebEx Teams to Microsoft Teams.

I've already created 3-5 POST requests which are working.

Now when I go to create another one (doesn't matter which alert it is in) I keep getting the error "Failed to execute HTTP request".

I am following This guide from Solarwinds which has worked for the 3-5 I have already created. I am using the sample text body from that guide.

What doesn't make sense is if this new POST request is exactly the same as the previous 3-5 I have already created, why is it failing?

Support sent me This article which wasn't helpful.

I feel like this is an issue with the Orion server and not the action I'm trying to create. Is there a limit to how many POST request actions a Orion server can have?

Find more posts tagged with

orion

post

post request

wenhook

webhooks

Accepted answers

All comments

christopher.t.jones123

@mfkdgaf, if you look in the C:\ProgramData\SolarWinds\Logs\Orion\ActionsExecutionAlert.log it should tell you what it's erroring out on.

mfkdgaf

Thanks, I found it.

The log shows the following.

System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.GetResponse()
   at SolarWinds.Orion.Core.Actions.Impl.SendHttpRequest.SendHttpRequestExecutor.ExecuteInternal()
2021-03-24 13:08:46,419 [7] ERROR AlertingLogger - (null)  Action [Action: ID: , ActionType: SendHttpRequest, Title: FAILING - (MS Teams) Alert Support Services, Description: Post to https://rush.webhook.office.com/webhookb2/random-string/IncomingWebhook/radom-string/random-string, Enabled: True, Order: 3 , Context: SolarWinds.Orion.Core.Models.Actions.Contexts.AlertingActionContext, EnviromentType: Alerting, ExecutionMode: Trigger, EntityType: Orion.Volumes, EntityUri: swis://ServerName.DomainName/Orion/Orion.Nodes/NodeID=128/Volumes/VolumeID=645, AlertContext: AlertName: RHA (Support) - Disk Space Alert, CreatedBy: , AlertActiveId: , AlertObjectId: ] execution has failed.
System.Exception: Failed to execute HTTP request ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.GetResponse()
   at SolarWinds.Orion.Core.Actions.Impl.SendHttpRequest.SendHttpRequestExecutor.ExecuteInternal()
   --- End of inner exception stack trace ---
   at SolarWinds.Orion.Core.Actions.Impl.SendHttpRequest.SendHttpRequestExecutor.ExecuteInternal()
   at SolarWinds.Orion.Core.Actions.ActionExecutorBase.Execute(IServiceProvider serviceProvider, ActionDefinition definition, ActionContextBase context, CancellationToken cancellationToken)

I have no idea how this is a bad request since my other working ones are exactly the same.

I've also tested by using the https://www.postman.com application directly on my Orion server and using This Microsoft article to post to MS Teams using the same URL which works.

So it makes be think that the body of the POST is incorrect but then why are the previous ones working but not the new ones.

Any ideas?

aLTeReGo

Could there be an API throttle limit on that you're hitting. This is fairly common these days. The throttle could be based upon total number of concurrent sessions open to the API endpoint, or limited per-IP, or even the number of API requests per-second/min/hour from that IP.

aLTeReGo

I would also suggest simplifying the post to plain text ASCI characters A-Z and 0-9. It's possible that there's an illegal character, potentially as the result of a macro being passed, that contains an illegal character that's not being properly escaped in your post request.

This (in theory) should be reproducible by firing the alert to an email or a text file, then copying the message contents and pasting it into Postman.

mfkdgaf

I've done some more troubleshooting and this is what I have found out.

If the trigger is alerting on the volume type then it will only work with SNMP, not wmi/SNMP.

So in my case only Linux machines. The rest of my machines are windows via agent or wmi/SNMP.

If the trigger is not alerting on volume type then it doesn’t matter how the system is connected to my Orion server.

christopher.t.jones123

@mfkdgaf, if you're alert is for Volumes then you are likely running into the issue described here. As @aLTeReGo mentioned this is caused by a value being passed that is not escaped correctly (in the article I linked it is specifically the Caption Variable).

chucksimalchik

I'm running into this with a different product (BigPanda) as well. I've successfully escaped the "\" in volumes, but am finding other issues with payloads. More specifically, we pass certain Windows Event Log messages over, and the receiving API rejects them. And in reality, there are other messages we just can't control in theory that may also prove to be problematic down the road.

I can simplify the message by not including those Event Log messages, but unfortunately we'll lose a significant amount of meaningful detail contained in them. I'm working on other solutions to try to sanitize the messages before passing them to the API, but certainly interested to hear if others have already invented this wheel before I re-invent it.