Alert aging and grooming

robert777

I am relatively new to Solarwinds, although I have worked with other monitoring tools such as SCOM and NetIQ AppManager.

Managing alerts:  There are now thousands of alerts in my Solarwinds system, some of them going back for months.

I want to implement basically strict alert grooming - anything older than 7 days gets purged.  No matter if it is open, acknowledged, no difference.  After 7 days it is done.

Looking at alert properties it appears that I cannot set a reset action on an "out of the box" alert, but I can put a reset action on a custom alert, say to reset after 604,800 seconds which is 7 days.

If I understand correctly this will make these alerts go away after 7 days.

I guess my main questions are:

1. Does the Reset Action really purge the alert after 'x' number of seconds?
2. Is there not a way to set an auto clear on out-of-box alerts?
3. Does Solarwinds by any chance have a 'global-auto-clear' script that can be run in the wee hours of the night to clear/purge everything older than 7 days?

Thanks in advance for any input from the community.

brscott

Yes, the reset after xxx time does actually purge the alert... but ... most alerts (especially from NPM and SAM) are based on persistent conditions and those will just come right back.  For example a "node down" alert happens because a server is no longer responding to pings.  You can clear that alert, but the node is still down, so the alert will just happen again... because the condition that set it off is still happening.  If you want to stop this type of alert, you have to "mute" the node or SAM monitor. (unmanage is an option, but that actually stops the polling for that object, which affects the graphs)

Out of the box alerts are nice to get you started, but personally I recommend that you disable every one of them, then duplicate the alert with a prefix like "AAA - alert name".  If the base alert is ever updated, your custom alert will remain untouched.  Now since it's a custom alert you can modify it, including the reset conditions, actions, etc...  Think of the built in alerts as a guide or template to get you started. 

No one really wants to be alerted on every issue.  You usually have a group of things you want to be alerted on, and a bunch more things you just want to monitor to have pretty graphs and data.  Then you have to decide how you want to distinguish between them.  Some people use groups, some use custom properties, some use the alert rules themselves with name matching.  I always prefer custom properties, they are more versatile to store information about production level, alerting hours, and responsible group, then use those custom properties in the alerting rules.

There are unfortunately a lot of ways to handle each situation, this is good for customizing it to your needs, but it also makes it difficult to say, "Do it this way".  Keep asking questions until the answer makes since.