Alert based on x number of polling cycles

Hello all,

I'm looking for a way to alertbased on x number of polling cycles. Currently we have a high CPU Usage alert sending emails based on its polling interval. Though in my environment i have servers that may get 2 mins of high cpu usage every once in a while but that is just normal usage that doesn't mean the server is having a problem. the way i can think to remedy this would be to set a condition in my alert that says send email after c # of polling cycles. Though i don't see one. can someone help???

Find more posts tagged with

advanced_alerts

advanced_alert

advanced_alert_setup_question

alert_condition

custom_alerting

active_alerts

Alerting

Alerts

Accepted answers

borgan

That is what the timer at the bottom of the Trigger Condition tab is for. Set it to the desired time interval. The alert will not fire until the condition has existed for that time period.

All comments

marunderwood

I'm still testing this, but you can give this a try. You'll notice I wrote a couple reports that I reference in the alert. I'm not comfortable with them yet, but you get the idea. Hope this helps!!

Future Need: We are currently having some db contention issues due to a new backup software app which is sharing the same storage server. So when the backups are heavy, almost every node is alerting. I'm looking into suppressing this alert when I hit a certain threshold (maybe 5 nodes), but I haven't figured out the sql for this yet.

Alert Name: (test) Managed node not polled during the last 5 tries
Description: Alert me when a managed node has not been polled during the last 5 tries (based on the node's polling freqency). This is different than the CPU polling (picks up more devices and it will fire quicker). Based on article: http://thwack.solarwinds.com/message/164931#164931 . 10/21/13 mtu Changed repeat from every 30 minutes to every 4 hours. Kept firing when the SW upgrade failed. Very annoying.
Check: every 5 minutes
Trigger condition (see picture)
Message Subject: ${AlertName} for ${Caption}.
Message Body:

${AlertName} for ${Caption}.
Node Id: ${NodeID}.
Node Status: ${Status}
Polling Interval: ${PollInterval}

ObjectSubType: ${ObjectSubType}
SNMPV2Only: ${SNMPV2Only}
StatCollection: ${StatCollection}
IP: ${IP_Address}
Comments: ${Comments}
Alert: ${AlertName}
Triggered: ${AlertTriggerTime}

To see all nodes failing, run report: https://(server)/Orion/Report.aspx?Report=Polling_Issues
For a historical report, run:
https://(server)/Orion/Report.aspx?Report=Node_Polling_Duration_by_Date_-_Last_30_days

Testing:
Skipped Polling Cycles: ${SQL:select (DATEDIFF(ss, Nodes.LastSync, getdate())/Nodes.PollInterval) from Nodes where Nodes.NodeId = ${NodeID}}
Last Sync: ${Nodes.LastSync} -- This shows up with just the time. The statement below includes the date too.
Last Sync: ${SQL:Select convert(varchar, LastSync,120) from Nodes where Nodes.NodeId = ${NodeID} }
Current Date/Time: ${SQL:Select getdate()}
Next Poll: ${NextPoll}
Last Poll Stat: ${CustomPollerLastStatisticsPoll} -- Not accurate
Last Poll Status: ${CustomPollerLastStatisticsPollSuccess} --Not accurate

Node missed 5 polls.jpg

thughes92

This is Helpful but im not looking for skipped polling cycles. My nodes are polling just fine.

I am however looking to send an email alert based on 3-4 polling cycles being over a threshold.

For instance, Solarwinds will poll a node every 60 seconds and every once in a while he will grab a High CPU Usage alert and email it. Then 60 seconds later the node is polled again and the CPU is down below the threshold. This Alert does not indicate anything except that the Solarwinds appears to be a pain in the rear by flooding several peoples inbox with Server Alerts.

The Remedy should be alerting on multiple polling cycles. say Server 1 stays above the threshold for 10 mins (x10 60sec. intervalls or 10 Polling cycles) this could indicate that after 10 mins of having a high CPU Rate that something might be going on.

Someone please advise with an answer

borgan

That is what the timer at the bottom of the Trigger Condition tab is for. Set it to the desired time interval. The alert will not fire until the condition has existed for that time period.

thughes92

Thanks a bunch! i completely missed that box at the bottom.