Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Prevent alert flurries during failover event due to service failure

Hello everyone,

Is there a way to prevent the inevitable flurry of false positive email alerts when Solarwinds fails over between our HA servers due to a service failure of any sort on the previously active polling engine?

Its super annoying to get pages in the middle of the night, just because a service on one of our SW boxes decides to fail for some reason (which has been happening a lot lately). I'm open to any set of recommendations/best practices that you follow, whether it involves SQL queries, API, or some other condition that I should add to our current alert "Node down" alerts.

Any help or information would be greatly appreciated!

Find more posts tagged with

Alerting

Accepted answers

All comments

Martin_Crothers

Are the alerts for Orion itself, or alerts on objects being monitored?

iteezy

Thanks for the reply.

They are alerts on objects being monitored.

jrouviere

Just was passing through looking for some other information, but having recently been working on our HA and having something similar:

What are the nature of your alerts? Are they node down alerts? Can you confirm that the secondary server can ping/SNMP to the down nodes if they are?

In our case we have ~100 nodes that our secondary server can't reach due to networking issues. So chances are this is a byproduct of the HA failover, but really it's a networking issue between the secondary server and the respective nodes. Resolve that and you'll resolve your alerts.

iteezy

So, as it turns out, this was due to the fact that on the majority of our nodes, we are polling via the Agent (agent-initiated) and not ICMP. When HA failover occurs, Agents are reported as down by the newly active secondary server because in their DNS cache, they are still reporting to the previously active IP address, even though the DNS A record has been updated to our secondary server. Once that DNS change propagated throughout our environment, the Agents could then contact the secondary server and were reported as Up once again. The solution to this was to change the up/down polling on the agent-polled machines from Agent to ICMP. Polling via ICMP instantly works after failover whereas Agent-initiated communication was delayed until the DNS record change was replicated to all of our internal DNS servers.