Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials

Alerting Engine / Custom Alert

I can't imagine we're unique in this problem but I'm trying to figure out the best way to solve this and I'm hoping someone may already have come up with a solution.

We have a lot of servers in Solarwinds SAM, around 1000+ and adding more. Many of these servers belong to different groups of people based on the applications they're running. The problem is letting the right people know when something happens. Sounds simple so far. We're using custom fields right now to help manage and categorize which works really great. We base a lot of alerts off of using the custom fields.

Here's where our needs get complicated and what I'm trying to solve without having to write a TON of similar alerts, This is kind of hard to describe to get the scope of this.

Dev needs an alert when something happens to their servers. Unless it's a SQL server, in which case that goes to a different team, but Dev still needs to know. Dev isn't production so we don't want the guys that handle production SQL servers to be notified unless the alert is a cluster fail alert, then they need to get involved. It gets more complicated when we get to the different Application servers that involve multiple groups.

So, right now everyone is complaining that they're getting too many alerts, including my group!

The idea I have is to take the alert and dump it to an external script, and let the script decide whom gets the alert. Is it a Database cluster down? Send and SMS page to the DBA group's phones unless it was a non-prod server. Low on drive space? If it's critical, send it to ops, otherwise send it to the app team that manages it. Do we have more than 10 VPN connections down at once, that's a problem, otherwise don't bother. Etc. Etc.

I'm trying to describe a message "router" of sorts that could take one alert that handles all the components of a template, and send messages to the appropriate targets depending on what is actually wrong. That way people only see actionable e-mails specific to them.

We have about 275+ alerts now. It would probably take triple that to try and carve them all up. There's gotta be an easier way.

Find more posts tagged with

Accepted answers

All comments

jm_sysadmin

Take a look here: Using Custom Properties sending Alert emails

It gave me a great start

kellydd

I'll have to think about it some more. Thank you and yes that helps but what I'm needing is more of a decision tree process, similar to the one that determines what gets alerted on. So when you create the alert, you say when "component x is down, server is type A, and server group is Group1, then fire." For example we have Database Clusters and File Clusters. All have the cluster component. Right now I have to build a separate alert for each cluster type, for each different division, for each different team within the divisions. That's a lot of alerts that could be solved with some kind of routing mechanism using custom property fields.

If I had a decision tree for the e-mail like "ServerGrp is member of "GlobalTeam1", ServerType is "Database", ServerClass is "Prod" then send to Address1, If ServerClass is "Dev" then send to Address2, If ServerClass is "QA" then send to Address3". Something like that. It would be more elegant. I end up with a lot of duplicate alerts that send the same alert to different people based on somewhat subtle differences in order to keep their spam down.

The reason the email custom property won't really work for me is where the e-mail needs to go depends on what the error is. If it's storage it goes to a different team then cluster problems, and a different team if it goes offline, app is over driving cpu, etc. Does that make sense?

tigger2

Sorry for the long post/typos..

The problem is, as I see it, is many-fold but starts out with the problem of you don't have the data in the monitoring systems(s) that you wish you had, even if you passed that data to a script.

The data fields used to generate an alert, and the data fields used to send the alert (or pass it to an external script) are the mostly the *same* data...so passing it to a script isn't going to help you (well sorta....more on that below)

I'll make up an example:

You have 2 servers. Both Windows OS.... both servers go down.... You want to alert to some people who can fix Windows "down" issues..

If you write your alert rules to try to route the alerts, you *may* have the issue that both are "Windows" server OS, but each reports as different versions of the OS (i.e. "Windows 2012 R2 Server SP2 enterprise" and "Windows Server 2014 R7, Platinum (c)").

So in your rules, if you want to catch it you have to put in something like "if osversion begins with Windows.*".. In your external script you would have to do the same thing.

Maybe you need to route it by Widows, and by prod/test/dev...same issue but you have to add a condition for that.

etc...

If I understand your dilemma correctly, I'd say:

- Put as much "data" as you can in Orion via Custom properties and let it *be* your database, so when you send the data to your script it has all (or much of) the data it needs to make a decision. For the example above, if you're routing by "Windows", then make a custom true/false property called "isWindowsOS" (Orion defaults to "No" on these) and then write a script to read all the internal data Orion discovers on the node and update the flag accordingly. Your rules/script logic gets a little simpler. Do this for the "big things" you need to route on, like environment (test, prod, dev, etc).

- Don't send alerts to individuals. Send them to "groups" or "teams". Make up a standard for naming the teams that starts with the application or service name and gets more granular as you read it like "Windows OS - support" and/or "Windows - Oncall - Primary". If you have to just send emails, then make the teams AD groups (if you use AD ). By doing this the people can move around and your rules don't change a lot. even if there is just 1 person in it, use groups/teams.

- Optionally: If you have some $, find an alerting tool to handle the routing of alerts to the members in the group once an alert it sent. Many have scheduling, rotation, and can do more than email. SolarWinds AlertCentral, or PagerDuty is an example.

With this set up your script or alerting rule gets a little easier as you can say something like: if isWindows = 1 and status = down and environment = 'Prod" then alert "Windows - Support"

However....

when you start getting into sub components and applications and "the details" you may run into the issue of "negation" in your rules. It's when your rules are all set up and a condition occurs and you end up notifying 12 people (well, teams) and not just 1. To do it with the above method you have to put in all sorts of logic in every rule like if isWindows = 1 and status = down and environment = 'Prod" AND host IS NOT 'abc' AND xyz IS NOT ...then alert "Windows - Support"

and things start to get really ugly. Here is where I'd possibly put negation conditions inside a script, esp. if the rules don't handle lists of negations/exceptions or regex conditions (like: host NOT matches" o.*mpx.*$") very well.

The only other solutions to this I can imagine, besides writing all the negation conditions in rules/scripts is:

- Your script (or something) has a database of alerts that it can query/scan for a time window (say 10 minutes) of all the alerts that have come in during that time (2 servers down 4 different applications alerted, and a UPS alert came in..etc), and then perform logic on that time window ( query what servers are connected to the UPS, it's the same two servers. Lookup applications dependent on these servers, it's 20 applications. Are the application alerts in that list...yes) to determine the root cause of the alert (power failure and... UPS failed?) and send just 1 alert (to the team that handles power issues at that location). You could probably even infer there are going to be 16 more applications reporting down very soon . If you can do this, I salute you. It's not easy, but can be done for some scenarios, but only if you have a place where applications and servers and network stuff and as much as you can are maintained in an easy to query way.

- Get people used to getting some alerts they may not have actionability on, or reduce the number of people expecting perfect alerting. Basically, don't allow it to get complicated if you can. An example I do is: Tell application people The "Windows - support" team gets CPU/Memory, up/down alerts. They will get with you if your servers are having an issue, so you *also* don't need to see these alerts (and have special rules written). Try to keep app teams just getting alerts for their app, not any of the extra things their app touches. It wont' work for all teams, but may cut down on complexity as some may go "ok, cool"

- Try to not send alerts out for certain environments, like test/qa.

- Instead of alerting try to make some reports that can be sent out or run by people on minor conditions ("disks are kinda-sorta full" report , etc.). Try to reserve alerting for critical alerts only.

- if you can find a tool that does it: Alert subscriptions. Basically: You get the majority of your core alerting to IT/Ops teams working as desired but you send a copy of all of the alerts to a secondary application. You let people who want alerts on all those extra conditions (like an app team who wants to know about UPS issues, for some reason) go in and write the logic themselves, so they get the alerts they want. It would be like allowing everyone to go into their own little Orion alert manager and write whatever conditions they want to see. Sure, all sorts of spam goes out but they spam themselves, and all the conditions people want to know about can be covered without you doing the legwork/maintenance. An example of this would be something like sending all alerts to slack or twitter.

- Oh yeah: a 24x7 help desk might be able just be able sort them all out and send/route the notifications manually. No scripts, rules needed other than "if x is problem, send to helpdesk"

jm_sysadmin

This is very good advice

kellydd

Good information! Thank you. I love the idea of an "Alert Subscription" where you could log into a page and select from a list of alerts which ones you wanted to receive. Kind of like a MajorDomo with a UI in front, allowing the alerts to be sent to one list and then distributed out. Interesting concept. I may look into building something like this. I don't think it would be that difficult.

We do let the Solarwinds database be our database and we use a lot of custom fields to separate and classify all the systems within. It allows us to query against it for server lists based on groups as well as our alerting. I feel we're pretty solid there.

I was just hoping for a better solution to where you could write one single alert, like High CPU Usage for example, and when it fires, it could go to any of a dozen or more groups/addresses based on who owned the server or who might need to see that particular alert. It's not that huge of an issue at the moment with only 1000 servers or so, but soon we'll have in upwards of more than 3000+ servers and many more groups to support. I'm just afraid of our alert "pile" becoming a bit unwieldy in its present state.

As far as our Monitoring center, I have built a single page console outside of SAM that shows all alerts that have fired on anything across the board, including custom alerts I've built to specifically fire and show up in the console but don't send e-mail. They usually just keep an eye on that page. I'll keep thinking and working towards a solution on this. Thanks!

tigger2

If it's of any help...

I had something small that I wrote as part of an existing application that did subscriptions. We didn't have a lot of options/features to work with, but the user UI was something like this:

Field 1: Hostname (drop down) [user input]

where: (drop down) were choices of: starts with, contains, matches, etc. Whatever was supported by the app. We had some simple regex capability.

Field 2: Severity (drop down1) (drop down2)

where drop down 1 were choices of: equals, does not equal

where drop down 2 were choices of only 4 severities: Critical, Warning, OK, and Information. Information was really for seeing things like people setting servers into maintenance mode.

Field 3: Team (drop down)

where choices of valid groups the *original* alert was being sent to. . You could pick only 1 group. It was useful for "listening in" on another groups alerts.

Field 4: Message (drop down)[user input]

where drop down were choices of: starts with, contains, matches, etc. Whatever was supported by the app. Also regex supported.

In addition:

1. The UI didn't allow people to put in things that would obviously result in lots of alerts. For example, you couldn't leave the hostname blank or put in .*. We simply disallowed < 5 characters as input...HOWEVER if you really wanted to you could use multiple %'s to get past it. It came in handy for admin use a few times.

2. There was a mini-viewer of all the alerts that were "in the alert system" so when you filled in the drop downs you saw what alerts you would get before saving it.This made it easy for people to build the filters, and for admins to go in and look at them as well to easily see how their filter was pulling in unwanted alerts.

3. The severe limitation on choices made it "do-able" for most people to set them up for themselves, but even then the admins ended up doing it for *a lot* of people. it turned into a way for admins to add simple alerting for people.

4. The severe limitation on choices made it usually necessary to build > 1 rule per person's request. Usually 2 would suffice. Per request.

6. For the Message field, for Orion alerts it would probably be the subject and body concatenated into 1 string, up to the system limit we could accept for our messages.

7. We had an "admin" level user that could subscribe alerts to any user *or team*. We used that a lot to build subscriptions that we didn't want users to be able to get into and change. Usually when someone wanted xyz alerts sent to the helpdesk. This was if we didn't want to make more Orion rules for them and the simple subscription would work.

8. *Most* of our core alerts went to about 4 core teams (Windows, UNIX, DB, Network), so if you wanted OS alerts, you just picked "Windows - IT" Team and then added hostname/regex and got them all. If you wanted just what waking up the on-call for that team, you added "Critical" as the severity.

The system we had had a few more pages to let the user determine how to alert themselves, and we only allowed them to alert themselves, not forward them to others. There could have been a lot more fields to use, but at the time we just didn't have a lot of reliable data we wanted people to have access to.

The major issues we had:

1. Hostnames. we had a lot of "well named" systems, but it didn't work well all the time. Our limited regex meant we had to make duplicate rules if a team had badly named hosts. And not many people wanted to deal with a complicated regex anyway even if we offered it.

2. We really wished we had an "AND" condition on the message field. Basically a second message field to filter on. To get around this we started adding special keywords at the end of the message (usually as exception conditions in our scripts before we injected the alerts) just to filter on this because we had only 1 shot to grab the message in the subscription. Usually this was for requests where the message field had a lot of data in it (like the location of a device, and the OS and they wanted x devices at y location). An example tag added would be "__SITE_NO_ZEBRAS__" or something, for those sites not surrounded with zebras (don't people ask for the oddest alerting?) because no one wanted to add an "Has Zebras" field to the monitoring system on all the hosts.

3. We did not have an "OR" condition, which would have been really nice.

4. Changing the message field messed everything up, this became an issue when we upgraded the monitoring systems and built in messages changed.

5. Over time a person would have all sorts of subscriptions...and...I really wished we had included a notes field so we knew why they were created.

6. Sometimes people would say "I want everything (insert person's name) is getting."..and that person had 10 separate rules and we had to build them for the person and hand copy them all. After a few of these were found we had to talk to the team and usually found an entire team had been building the same subscription, for each user, over a period of time. We simply built them a new one and made them an "official" team, or made an Orion level alert rule for them, for example.

After it was all said and done, I believe it would have been wisest to NOT have included a message field. it was done because it was an easy solution, and a lot of data was in that field. People would usually get an alert and email it to others, who would then day "I want these too", and it turned out the logic they wanted was somewhere in the message field. If I had to do it over again I would probably only allow drop downs (and maybe some regex's here and there) of carefully curated data fields.

sacosgrove

I have a similar issue and here's what I did leveraging custom properties:

I created a new custom property for each team and labeled it TeamX_Node (yes/no). I then decide which nodes I want to be visible to that team and set the value to yes.

When I add TeamX active directory group for access to SolarWinds, I apply an account limitation for TeamX_node. I set the alert and report limitations for TeamX.

Then I duplicate and edit an existing alert, I set the alert limitation to TeamX, the responsible team to Team X, the scope of the alert trigger to TeamX_Node is Yes, and change to email recipient.