Can alert trigger conditions be added to multiple alerts at the same time?

I just added an custom property called "environment" to all of the nodes to separate development and production systems, but our existing alerts do not check the environment as a condition. so non-production nodes are causing alerts to go to our on-call staff.

I'm looking for a fast way to add a condition of "environment=production" to all of our alerts. Is there way to do this using SQL or the Web console ?

We also considering adding "operational status" custom property so that nodes in a build or maintenance status will not cause alerts either.

Find more posts tagged with

Alert

production

Accepted answers

All comments

mesverrum

Sadly there is not a method in place for bulk editing the alerts like that, you are going to end up having to edit them 1 by 1 in the GUI. Going directly to the SQL would likely take so long to develop and have a relatively high risk of failing that unless you have somewhere over 100+ alerts to configure then you won't come out ahead. And in the case that you actually do have that many alerts you should probably spend some time re-evaluating your custom properties and alert logic because if you are setting them up correctly you should be able to cover just about every use case in under 30 alerts. Probably much less, depending on how many modules you are running.

-Marc Netterfield

Loop1 Systems: SolarWinds Training and Professional Services

LinkedIN: Loop1 Systems
Facebook: Loop1 Systems
Twitter: @Loop1Systems

FormerMember

I recently inherited the system so I am in firefighter mode figuring out how to get the kinks out of the existing system while developing best practices to have system meet requirements in the most efficient manner possible. The number of alerts that exist and the lack of documentation is daunting. On the other hand, if I get things under control I'll come out a shining knight!

Thanks for the advice.

mesverrum

When I find myself in that position I usually end up just spending a day or so going through the existing alerts, taking notes on figuring out what they were intended to do and starting to plan out how to condense/improve them. Then I build my replacements and give them all the same prefix like !Alertname so I can find them all in one place. Once I think i have every alert scenario replaced and, if possible, tested I just enable my new batch and disable the old ones. The nice thing about this method is it keeps the old alerts intact in case there was something i didn't catch and I need to go back or turn one of them back on in the future or use it as a reference.

-Marc Netterfield

Loop1 Systems: SolarWinds Training and Professional Services

LinkedIN: Loop1 Systems
Facebook: Loop1 Systems
Twitter: @Loop1Systems

FormerMember

That makes sense. I'm going to be cramped for time with all of the upgrades, fixes, and just plain learning I need to do., but I'll use your idea to rename the old alerts and then come up with a new alert naming and design pattern to make them easier top manage going forward.

I'm don't know which alerts are valid and which ones were created for experimentation by people involved with the transition before I got that app, so there is that obstacle as well.

My goals are to get all of the systems and devices into monitoring,

upgrade the software that is being on patching

identify performance constraights, which it's apparent from timeouts and slow loading web console that they exists

get the alerting sorted

and finally, make the system as self help as possible. I'm in a very large and varied environment , so support people will know their app/system much better than I will ever know it. I'll get the system up and running with good design , policy, templates, and documentation. I'll let them do the threshold and alert building.