Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials

New to Orion - need help with alert email addresses

New to Orion. Have been put in charge of migrating our old monitoring system to Orion. I have most tasks done (it was mostly set up when given to me) and I have all our systems in one group, with subgroups for easy reference. I also have all systems being monitored to some degree successfully. But now it's time to work on the alerts, which are also mostly written. So I'm not looking too bad at this point.

Here's the issue. We want all alerts to go to an email group. No issue there. I can get the distribution group created and use that in the TO: field on the alerts. However, we also have a rotating person on-call. Depending on the person, they will use either their personal cell phone or an issued pager to receive any alerts while on-call. The issue then becomes, how do you change the address when the person on-call changes in all the alerts? I think it would be the simplest to put that in the CC: field. But with many alerts, I'm looking for a way to change all alerts with just one task. (I.E., change it in one place for simplicity and it will effect all alerts.) I'd really rather not have to tell the person when they go on-call that they have to go in and change ALL the alerts to include their number/address.

I kicked around the idea of creating a custom field, but that would effect all our nodes (120+) and be a worse solution because all the nodes would then have to be changed. Then I considered creating an unused/fake node and putting the address in a field for that node. But then the question becomes, how do you reference that field, on that node, in the CC: address in all the alerts? I even considered creating a report (somehow) where the on-call person could bring up, change the report and have it do a SQL query to update the alerts. Unfortunately, being new to Orion, this is over my head at this point.

Does anyone have any suggestions how I could streamline this?

I'll gladly give points once I figure out how to do that too. (This is my first post to the community!)

TIA!

-Gonzo

Find more posts tagged with

Accepted answers

All comments

byrona

Would it make sense to also setup a distribution group for OnCall and when the on-call person changes go in and change the distribution group? This would give you a single point to make changes when the on-call person changes.

great.gonzos

Thought of that too. Unfortunately, as the UNIX team, we don't have access to the DG's in Exchange. We have to coordinate that with that team. That's why I can get the DG for the bulk of addresses, those that won't change done. But for the on-call person, we would be requesting a change to the on-call DG weekly - something they AREN'T going to want to do. So I'd like to handle it internally to Orion if at all possible.

SKerley

We use exchange and do what 'byrona' suggested. All production alerts go to systemsalert@mycompany.com. In Exchange that email address points to who ever is on call's email address first.last@mycompany.com. Who ever is on call can just go in and add their email address to systemsalert and take out who's not. Of course, we just leave all our addresses in there and just don't take home our BB if we're not on call. Or turn the ringer off.

great.gonzos

Like I said, if we had access to Exchange, we could do what SKerley (and byrona) suggests. But I have to do this without an Exchange component. I can use an Exchange DG for the main bulk of addresses that won't change, but for the person on call, we can't change or request a change to the addresses or DG every week. That's why I asked for a way to do this within Orion. That we have access to and can make all the changes we want.

netlogix

in exchange you can delegate control of a distribution list to a non exchange nor AD admin by setting them as an owner.

aLTeReGo

Many of our users who have on-call rotation use PagerDuty. It's probably worth taking a look at.

phillipmanning

As mentioned before, the best solution would be to set up DLs that the users could modify and you would use the DLs in your alert. An alternative might be for your users to maintain one or more csv files containing the appropriate SMTP addresses that is accessible from your Orion server. You would then need to invoke a script from the Alert rule that would read the file to pull the oncall person’s contact info and send the email to the address(es).

great.gonzos

All - I appreciate all your inputs. The common answer is to use an email distro-group that the person on-call can change when it's their turn. However, as I said, we don't have that kind of access, nor will we get access to change our own DG. So, that solution, while I agree is probably the best overall, if for no other reason than simplicity, it won't work for us.

That being said, I have found another solution that turns out is already be in place. I found it by talking to our lead DBA as this is how they've done it for their group. As we have many groups using our installation of Orion (UNIX, DBA's, Windows, network, infrastructure, etc), they each have their own alerts and their own person to maintain their tools (monitors, alerts, components, notifications, etc.). The result is that the alerts all start with a significant word to distinguish to what group they belong (to help keep the UNIX guys from modifying the Windows alerts!). The DBA alerts all start with their own word ("ITDBA" for example). Our lead DBA told me that they've been doing this for a while. The UNIX alerts have always started with "UNIX", and I found out, that was the wrong significant word.

They (the DBA's) run a script (SQL) on Monday morning that does a couple of things. First and foremost, it queries our on-call schedule to determine who is on call. It then looks in the Orion DB for any alerts that start with "ITDBA - " and then modifies the "TO:" address field in the alert to the new on-call person's phone's email address (9999999999@phoneservice.net). Secondly, it also creates an email and sends the two people involved (the one going off on-call status and the one coming on) to remind them of their duties.

So in the end, all I really needed to do was change the name of our alerts (the "UNIX" alerts) and talk to the lead DBA. Since we've never done it this way for the UNIX guys, while the SQL code was written, it's never been used (it has been commented out). So it just needs un-commenting in the script and it will do the same for our alerts as it does for the DBA's.

I created a test alert and we are scheduling a test to make sure it works. But from what he told me, there shouldn't be an issue as they've been doing it this way already.

While I agree that this isn't the simplest solution, it works. It's also automatic in that the changes needed are automated.

Thanks for all your help! It's good to know that a you guys are so helpful!

-G

FormerMember

Hey Gonzo,

To add to all the suggestions, we are building a product that handles on call and escalation policy routing for alerts. If you're interested in beta testing, providing feedback, or even just hearing more about it, we'd love to add you to our list. It sounds like it might be able to automate what you're looking for (well, once we release it ).

You'll be able to define on call schedules and escalation policies (notify the person on call, then the secondary on call person, then try contacting the group) and create different policies for different sources/alerts (if the alert from Orion contains "ITDBA" route it to DBA group, if it contains "UNIX" route to UNIX group; if any alert from Non-Orion-Product comes in send it to the Windows group).

You can read more on our earlier thwack post:

Thanks for sharing your solution, too.