Help on creating a special alert,

Hi currently we are using custom properties for our smart alerts. we have set up one alert 'Any Node Down'

when a node is down it then looks for custom properties in trigger actions to send email to people that are concerned. If the custom properties are filled in the node details then people get alerted.

We recetnly have ad Service now as a Helpdesk tool. Now when anyone emails the service desk it automatically gets logged as a ticket.

On Orion custom properties we have put down the london service desk email address for some of our servers, so that this gets logged as a ticket for tracking purposes.

We had an incident where one of the WAN's was intermittent was was going up and down intermittently every 5 minutes over the weekend. This obvioulsly logged around 400 tickets.

So what we now want to do is set up a special node down notification and we want it to do the following.

When a node goes down a notification is immediately sent to the ServiceNow address.
Once the first notification about the node being down is sent another will not be sent to this address for an hour (if it is still down).

What i have done is created a extra custom filed called techinicalgroupSN. I have then created an extra Tigger Action and have put down the servicenow email address. In the 'Alert Action' I have delayed the action by one hour just for the servicenow email address.

What this is doing is if any node is down, it will send alerts to all the people in the custome properties staright away. I have excluded the servicenow email address. The second trigger action waits for one hour and if node is still down then it will send email to servicenow which then creates and ticket. This is almost there.

What i need it to do it....send an email immideiately to servicenow and once first notification is sent about node being down another will not be sent for an hour (if still down). I know there is a way i just can't put my finger on the issue.!

I have attached screenshots of my current Alerts and the way they are configured.

Alert Screenshots.docx

Find more posts tagged with

Accepted answers

All comments

popearce

Rupen,

I'm not sure why you want or need to send a notification in an hour. If you're following ITIL, once an event has occurred and a service is affected/down, you want to create an incident and begin the incident management process. Email ServiceNow upon status change (service affected/service resumed) so ServiceNow can create an incident or resolve an incident. ServiceNow has extensive email processing and scripting capabilities, much more than SolarWinds, to handle your alerts in any manner you wish. Use each platform according to their strength. Stop using SolarWinds for incident management and do use SolarWinds for event management.

#ServiceNow #integration

Paul

For bonus points, you can create and iFrame of your SolarWinds Event Manager within ServiceNow (similar to the Dell KACE integration).

netlogix

You could create a separate alert that has a reset time of an hour (in alert open the Reset Condition tab change the "Do not reset ..." to 1 hour - if the radio button is set to reset when no longer true, you'll have to change that and set the reset trigger to something like status = up).

That alert will stay triggered till the reset condition exists for more than an hour continuously.

I know a 2nd alert isn't always the thing you want, but sometimes it is needed.

rupen

Hi Paul,

in your resposne you said

'I'm not sure why you want or need to send a notification in an hour'

we need to delay it due to Servicenow being spammed with emails when there is an intermittent issues. We expericned this with one of our sites when WAN was down intermitently iover one

So what we Orion to do is.

• As soon as the node goes down an notification is sent immediately.
• No further notification will be sent unless the node is down over an hour in which case a notification will be sent each hour.

rupen

Thanks for your reply,

Yes we have added a 2nd alert as you mentioned. Howevber this is a problem, as we added the london helpdesk email on the 2nd alert. So when its down it will not send email to london helpdesk unless its down for more than one hour. What we want to do is send one email when its down, then if the device comes back up and then goes down within the hour then DO NOT send email. Only send further email if its down for more than one hour. So if a device intermintently goes down say over a weekend then orion will only fire email for london helpdesk no more than 24 emails as opposed to 400 that we had over one issue on a Saturday.

Thanks in advance for everyones help.

Rupen

rupen

Hi Paul,

I think I may have been unclear on the request. We are following ITIL. So what we want to do is…

· As soon as the node goes down an notification is sent immediately.

· No further notification will be sent unless the node is down over an hour in which case a notification will be sent each hour.

I have managed to delay the notificatio0n by one hour. So its kinds of half done.

Why we want to delay for an hour? Well we had an incident few weeks back where a WAN devices kept going down intermittently. Now because we have added London.helpdesk email in one of the custom fields. On that weekend Servicenow Logged over 400 tickets. For one device down.

We want to counteract this problem by configuring the alert so when its down it will email London helpdesk, but if the same device is down in the next hour it will NOT send an email. So in fact it will send email every hour after that.

I hope I have made sense, if not I am more than happy to discuss this with you ?

Rupen PATEL

Enterprise Application Support Specialist

Tel: 0044 208 326 5020 | Mob: +0044 755 400 2822| GMT +0

popearce

Rupen,

I did an analysis for a major telecom that was flooding ServiceNow with alerts from Orion. They were using ServiceNow for event management and incident management.

When an alarm was triggered in Orion, an email alert was sent to ServiceNow.
Every alert from Orion created an Event record in ServiceNow
The first Event created an Incident, subsequent Events were added to the Incident record.
When a reset alert was received, the Incident was marked as "Resolved".
In this way, only one incident was created per outage.

Problem was, they were sending alerts ever 1-15 minutes, which was creating millions of Event records per year. I analyzed all the event records and determined that 94% of the records were merely updates, without providing any new information or status. That is to say, 94% of the alerts/Event records just said the device is still down.

What I suggested to them and I am suggesting to you, is this:

Send an alert to ServiceNow ONLY upon a change in state, such up/down
ServiceNow checks to see if an active Incident exists before creating an Incident
A reset alert sets the Incident record state to "Resolved"
Alerts from Orion sent to ServiceNow while an Incident is active (any state other than "Closed") are suppressed, no new Incident records are created
Additional alerts received while the Incident state is "Resolved" could change the state to "Flapping" or "Intermittent"

In this way, SolarWinds is used for intelligent alerts and Event Management, and ServiceNow is used for Incident Managmeent.

I hope this helps.

rjnicholson

Create those nodes as a dependency of the WAN link using Groups and Dependencies, so if the WAN is bouncing and dropping out it's known that the nodes behind that WAN are dependant on it being up for them to function and properly report polling status. Once you do this the nodes behind the WAN become "unreachable" and won't trigger any of the alerts you don't need when this WAN link is down or bouncing.

I know this isn't related to the way you are setting up your alerts and escalations of alerts, but this will help eliminate false positives in alerting on device behind links that are critical to them being up. After all if that link is down I would imagine that your Support Center would know that everything behind it is down a well, and they don't need alerts for them at all.

rupen

I would like to thank all for their input, at least now we know the capability of Orion and now can push Servicenow for further enhancements. Thanks all!