In what ways have you automated NPM alerts to give you only the pertinent data when a router goes down?

Question

When one of our WAN sites stops communicating with our home office, it can be a life-threatening event, or a business-affecting event, or an expensive inconvenience--since we're a group of a hundred hospitals, clinics, business offices, and warehouses.

During business hours we see the nodes at the site go down, or just the site's router down (because of Groups and Dependencies).  But we DON'T see all the info we need to proceed.  Instead, the alert displays the node name (which includes a SITE ID abbreviation that can be confusing), the node's management IP address, and the time of the outage.

I'd also like to include:

* Start of the outage
* Intuitive site name
* WAN provider name
* WAN provider outage telephone number
* WAN provider circuit ID, Billing ID
* Site physical street address
* Local site hours
* Local contact (Receptionist, engineer, IT person, etc.) with phone or pager numbers

This information is available from several different web pages, and also from a WAN document.

Ideally NPM would have a method of using some primitive AI to gather all this information and display only the appropriate information in the Alert.  And it would be enough information that the alert could be sent to a Network Analyst, a Help Desk mail account, or a WAN provider service outage e-mail address, and everything necessary would be in the alert.

How do you configure alerts & nodes so you see everything you need in an alert?

Swift packets and my sincere thanks for useful examples that show how to do this.

Rick Schroeder

superfly · Answer

In a previous role we just used custom properties for this information. We had details like link ID, provider and phone number, site hours and contact details etc

Then in the email we just laid it out nicely and added in the custom properties. So the outage email had all the information you needed without having to look elsewhere.

dark235 · Answer

The best way I could suggest you do this is to have an excel sheet as your master circuit list that is maintained by your Circuit team that contains the provider, provider contact, CircID, site name, address, hours, contacts, etc.  Use SQL to parse the Excel sheet info and drop it into a custom fields in Solarwinds.  Maybe that Sproc runs once a week and just updates all of the fields or maybe you use something like Zapier to trigger it off a shared document.  Either way, dump all of your metadata into Solarwinds custom fields.

Then you just have to set up an alert to pull the custom fields when/where appropriate.

jrouviere · Answer

To me, this looks like a clear cut case for Custom Properties.

I'm sure it's going to take some time to gather, but how dynamic is this information? The only thing you've listed that I see would remotely be dynamic would be the site contact (could depend on the hours, days, etc, especially with the holidays coming up).

Everything else is static or relatively so and a few of them we have similar properties for in our own environment.

Not to say that you couldn't spend the time to drum up some script to access your various websites to get this information, but when is that information going to be gathered and where will it be stored? If it's run as part of an alert, your best bet is likely to store it as Alert Notes in a similar way as the SAM alerts for top 10 processes. Otherwise you would run the script periodically to update the Custom Properties in any case.