SolarWinds - Updating Alerting within ServiceNow

RE: SolarWinds - Updating Alerting within ServiceNow

Hi,

Seeing if anyone out there has the same setup as the company I work for, and either having the same issue or has had it and resolved it.

We have a number of SW Modules including the key ones (NPM,SAM) etc and currently running version (2020.2.6 Orion HF5) so up to date. ServiceNow is up-to-date just not sure of version.

As many do we have alerting in place between SW & ServiceNow, but this is where we may differ from many places we don't have it set up the standard way. Solarwinds doesn't send the alerts to ServiceNow and a Incident gets created. We have ServiceNow talking to Solarwinds.

We are using the NetPerfmon log option. So ServiceNow scraps the events in SolarWinds. Certain Fields are mapped across from SolarWinds in to ServiceNow. This in turn creates the Incidents and from the action you see below certain variables within this will populate a number of fields such as Short description/ main description etc.

This hopefully gives anyone a basic idea who may have it set up in this way. Appreciated their is more to it, but trying keep it simple as possible.

Alerting is currently working, but we are trying to make a change which only partially works. This is where is starts to get complex so I will do my best to explain it in simple terms.

We are trying to ensure Incidents within ServiceNow are updated within the same incident and closed off correctly (This includes SolarWinds Alerts).

Example Alert we are testing :

Volume Alert

1. SolarWinds P3 Warning Alert created.

2. ServiceNow Incident created.

3 SolarWinds P2 Critical Alert Created.

4. ServiceNow ticket updated.

(So Far all works)

5. Engineer fixes the P2 critical Volume issue. SolarWinds P2 Critical Alert closes,but so does the Incident.

(This still leaves the warning SolarWinds Alert open. Meaning we could have stale alerts left open and worse preventing further alerts not being created until these have been cleared).

We initially tried to put a trigger/reset action but this failed. After looking at this further we could see each of the alerts create in Solarwinds have different Event ID's.

Hopefully you can see better with example below.

Disk P3 Alert
- Warning -> swEventId=12345
- ServiceNow Ticket 5999
- swEventId=12345
Disk P2 Alert
- Critical -> swEventId=67891
- ServiceNow Ticket 5999
- swEventId=12345
- swEventId=67891 <====== at this point the P3 eventId reference is overwritten in ServiceNow

(Engineer sorts the P2 Volume Issue)

(scenario) P2 Alert automatically closed
- workflow closes swEventId=67891
- P2 Alert closed in SolarWInds
- P3 Alert stays open

We are looking to see if we can add this addition ID in to ServiceNow as an additional task, but it's then assuring the closure process is correct as well.

I appreciate I have kept it simple,but if any body has had a similar setup and has come across anything like this similar I would Massively appreciate if/how you may of overcome this.

Happy to provide more detail if anyone can shed further light on this. Maybe something we have missed.

Thanks.

Find more posts tagged with

Accepted answers

All comments

Seashore

I haven't seen an integration like this before but I guess here it's more about that the alerts in Orion is not closed as they should, right?

So for example.

If DiskUsage>80% THEN P3 Warning
If DiskUsage>90% THEN P2 Critical

Solving the P2 and disk usage is below 90% the alert will be reset. Same goes for the P3, if usage is below 80% the alert should be reset IF you are using the default Reset condition on your alert: "Reset this alert when trigger condition is no longer true (Recommended)".

scottm78

Yes that's the idea of it in regards to P3/P2 for example.

We have tried a trigger action to only reset it when the warning has been cleared as well. This was to try and ensure that the INC as well as both alerts would be closed. Sadly this doesn't work. this also goes for the reset trigger as well.

This is when we realised its down to the separate Event ID's that are generated from both SW alerts. So as soon as the P2 triggers this then updates the ticket and overwrites the previous event ID with the new one. So there is no further association with the P3 alert.

This is why it seems that the warning alert wont clear. We have spoke to one of out engineers who has a good understanding of ServiceNow and we are going to see if we can see about associating the first event ID with the updated ticket.

This will close off all the alerts when one of the events is closed, So When a P2 is closed in SW and in return sends a closure transaction to ServiceNow, the Alert and Incident both will be closed and will use the existing flow of close all the open events associated with that incident in SW. This will require of finding a way to store multiple Event ids on the same incident and a filter condition to trigger the flows.

So it's how straight forward it would be to do this. Also how reliable & efficient it is. As you know yourself its trying to automate it and have it as slick as possible. Appreciate this isn't the norm for most company setups.

marlief22

Hey there,

Based on your description, would this work:

P3 Trigger:

P2 Trigger:

P2 & P3 Reset:

In this configuration, the alerts will only clear once the Critical AND Warning values are no longer met. It also allows you to set individual thresholds for each volume (Allows better scaling later on)

Let me know how you think and if you have any questions.

scottm78

Appreciate your feed back.

We have done something very similar in our test environment for both trigger & reset, but this only closes the P2 Critical.

In our production currently we have separate Incidents being created for a P3/P2 etc. So they close when resolved as expected.This is because they are sperate and have separate IDs.

We have changed this is test. We are now creating one Incident and updating that incident within ServiceNow. This has changed the dynamics to it now. We still have two Event IDs but because the INC is being updated the P3 Alert Event ID from SW is being overwritten by the P2.

So even though we put a trigger/reset action in. It will close the INC off, but only close of the P2 alerts as it only knows about P2 Alert as the P3 no longer has any association to it.

So it makes things more complex and confusing as are doing it the opposite way with ServiceNow than most. We are also updating the INC. where I'm guessing others will create separate INC from the SW alerts.