Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials

Alert ONLY on >x failures?

I have some remote sites and devices that are flaky and will come back up on their own, reboot independently on a schedule, etc.

My current monitoring app I use has the ability to only alert after x consecutive polling failures. That way if its a self correcting transient issue I dont get alerted for every little glitch in the system. For example, if a remote admin reboots a router, the system sees the router down on pass 1 and does nothing but log it. If it sees it is still down 90 seconds later during the next polling pass, it will then take action and page me. It still emails the individual alerts to a mailbox so we can see potential problems such as a bouncing circuit, but wont alert us at 2am just because a device rebooted during a polling cycle.

Is there a way to only alert (in our case send a text message) to the recipients if the outage is longer than x minutes, >x polling failures, >x SNMP Traps, etc?

Find more posts tagged with

advanced_alerts

Accepted answers

All comments

mmelton

On the Alert in question go to the Trigger Condition tab and at the buttom you will see "Do not trigger this action until condition exists for more than: Here you can set the time it takes to trigger the alert. But if you want this on only certain nodes you will need to add a custom property or some other unique identifier to which devices fit this need, and which you want immediate notice whenever it is detected as down.

cameramonkey

That solves most situations except one.

We have a fiber link with a backup T1 failover. if I set the alert for that particular interface to alert after x minutes of sustained traffic, wont that possibly extend the delay indefinitely if the circuit doesnt immediate saturate and stay pegged out? If it drops below the threshold repeatedly I could see it taking a long time to alert.

For example if it starts passing traffic and the threshold is >5% and the utilization drops below 5% during the wait interval wont that drop reset the counters and not alert so that it would possibly alert after numerous drops below the threshold and only alert once the traffic is sustained for more than the wait interval?

Or am I over thinking things?

mmelton

So in this instance you are worried about the alerts on your backup circuit? Are you wanting this alert to know when your main circuit has failed?

cameramonkey

Yes and no. we have some OSPF/BGP routing issues where our failover kicks in for no good reason and has our carrier stumped.

We cant actually test for the fiber dropping because everything still responds to pings... the traffic routes just change. A simple BGP reset command on one of our routers and its back to normal, but we need to be alerted to the traffic so we can issue the command. For a big reason I cant recall at the moment we cant auto fail back so we must do it manually.

So from a device/interface up/down perspective, the fiber doesnt actually go down so we have to be creative in monitoring for interface loads over device status.