Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials

Flood of Alerts on Each SolarWinds Reboot

We have the SolarWinds Platform installed on a single server. Any time that server is rebooted, it shows all servers as down and sends a flood of alerts to all the admins. Is there a way around this? With 600 servers that is a lot of emails to process.

Find more posts tagged with

Accepted answers

All comments

bourlis

If I understand correctly, your SolarWinds is sending out 600+ node down alerts when you reboot your SolarWinds server?

If this is correct the problem is most likely the node down trigger condition for the 600 servers. I would double check the scope of alert and make sure that it's not set to All Object in my environment. Set it to Only following set of objects.

Next, look at the trigger conditions, make sure that is says Node Status is equal to Down.

You can add <or> Node Status is equal to Unknown. But I wouldn't!!!!! Keep it simply get it to where you like it and then build from there.

Your reset condition should look the same, except set Node Status is equal to Up. Anything else will cause more problems.

oithim_yew

First, check what are the 'Status' of the nodes when the alerts are triggered (usually it should be 'Unknown')

Second, I would specify the down alert with the "Status==Down' to specifically specify the status of the node

Third, I would create another child conditions for that with much more specific conditions.

Example:

jduehmig

Thanks for the response. The existing alert is set to "is equal to" "Down", and must exist for more than 5 minutes. The agents themselves would know that the servers are not actually 'down', so why does the server generate the alerts when it is rebooted?

jduehmig

Anyone?

bourlis

There's lots of different ways to tackle it, but here's my suggestion, do a litness test in SWQL Studio. You can test it all day long in a controlled environment that's not service impatching.

Create a custom property called 'Alert_on_Test'. This will be used as an analog for testing that'll replace the node status.
Now in SWQL Studio use enter the following script.

SELECT n.DisplayName

FROM Orion.Nodes AS n 

WHERE ((( n.IsServer = 'True') 

AND ( .[IsOrionServer] = 'False' ) ) 


AND ((n.[CustomProperties].[Test_on_Alert] = 'Alert_now')))

Now set the CP Test_on_Alert on a server that's not a SolarWinds Server and run the query. This will simulate being down and you should 1 return from the query.
Now set the CP Test_on_Alert on a server that is a SolarWinds Server and run the query again. This will simulate that server being down. The return should not increase; it should still show the first server.
Next, I would copy the original alert, modify the alert so that it matches the SWQL query, set the trigger condition so that it only emails you and then rerun the test. Your new alert should look like this:

If the results are the same as the SWQL test, then you can move forward and change the trigger condition to status = down. I suggest you leave both alerts in place for at least for one Windows patching cycle. If this resolves your issue, then edit the original alert to match, delete your test alert and delete the CP.