Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Unmanaging and Managing Nodes Causes multiple alerts to be generated

I've noticed that when I un-manage and then re-manage a node I get two alerts for each node. Un-managing the nodes works as expected, but when I go to re-manage the nodes I get two alerts, one showing the node as "unknown" and the second showing it as "up". I also noticed that if I do this to a lot of nodes at once, for quite awhile after the nodes have been re-managed the status shows a grey color for all applications and it can take up to an hour for all of them return to green.

What we are trying to accomplish is a way to silence the thousands of alerts we get during patching and network maintenance windows. I have implemented the n_mute way of quieting single nodes mentioned in another article, but there seems to be no way to "mute" all of the nodes at once easily. I have also tried stopping SAM services and turning off the server (crude I know) but we need to be able to schedule this on and off so we don't have to get up in the middle of the night to do it.

Would Alert Manager be a better way to try and manage alerts during change windows? I've tinkered with it, but never really tried to put it in production.

Find more posts tagged with

Accepted answers

All comments

adatole

Yes this is how it works. It makes sense if you think about it - coming out of a "blackout" you would want to re-test all conditions to make sure nothing changed while the device was invisible to monitoring.

However, here's my way around it:

http://thwack.solarwinds.com/message/142288#142288

aLTeReGo

I recommend excluding "unknown" from the alert trigger condition, this will prevent you from receiving the "unknown" and subsequent "up" alert notifications. If necessary, create a separate alert to notify you of "unknown" status only when the condition exists longer than "X" number of minutes. What's occurring when the node/applications(s) are remanaged is that polling jobs are scheduled, but the Advanced Alert Manager immediately begins checking their status which obviously hasn't been updated in possibly hours.and finding that there is not status for those objects. At least not until their next poll completes successfully. As for applications, if there are a large number of applications that are suddenly remanaged the Job Scheduler must spread these jobs out such that the poller is not overwhelmed. Depending upon the number of applications and how many components are contained there-in, their type and responsiveness, can take some time before all of them are successfully polled and properly scheduled in an even fashion.

jason1320

I use the n_mute but I have only changed the attribute one server at a time. How do you change the properties on all of the at once? Custom property editor? If so, is there a way to schedule it?

jason1320

That information is very helpful. I'll remove "unknown" from the alerts and see if that solves the issue. Thank you for the quick response.

vinay.by

1. Why don't you use Unmanage scheduling utility ? You can unmanage all nodes/interfaces/applications at once as required (during your patching schedule) - You would find this utility on your primary poller.

2. If you have a regular fixed timeline for your patching, you can go with this option. You can as well create a scheduled task on your server for the same (use windows task scheduler to run your unmanage schedule file).

3. Wrt to the next problem you have with unknown's - looks like a lag on your poller. I can only suggest to use a time gap while you re-manage the nodes. But then it depends, if it isnt a great option please don't go with it.

Say you have 1000 nodes to re-manage, dont load your poller by re-managing all of them at once. Break it into small chunks like 10X100.

When you create a unmanage schedule, for every 100 nodes have a 5 min gap or less - this might reduce the unknown count.

Hope it helps

jason1320

Update: Our SQL DBA wrote a stored procedure that flips all the n_mute attributes on and off and then wrote a little web interface for it so the admins can set the schedule. Currently it's all or nothing but he's working on a grouping function for us. Once nice side affect of this method is that everything is still monitored during the outage window it just doesn't email and when the email function is re-enabled any alert that is outstanding will alert immediately. (Disk space, component down, etc.)