Simple Alert Needs - Help!

So we have some simple needs. We implemented two alerts as follows - both with email actions for triggers (Node Down, Node Rebooted) and reset (Node Down):

Node Down - We want to know (email) if a node goes down - after it has been down for 5 seconds. We then want to know that the node has come back up - after it has been up for 15 seconds.

Alert Rule - Runs Every 1 Minute

- Trigger - Node Status is equal to Down - 5 seconds
- Reset - Node Status is equal to Up - 15 seconds

Node Rebooted - We want to know when a node reboots.

Alert Rule - Runs Every 1 Minute

- Trigger - Last Boot has changed - 0 seconds
- Reset - Trigger is no longer true

Here are the alert logs for a test node:

Node Down  5/5/2010 9:12:49 PM TESTNODE has stopped responding (Request Timed Out)
Node Rebooted  5/5/2010 9:13:46 PM TESTNODE rebooted at 5/5/2010 21:11
Node Up   5/5/2010 9:13:46 PM TESTNODE is responding again. Response time is 1 milliseconds
Alert Triggered  5/5/2010 9:13:57 PM Node TESTNODE has rebooted at Wednesday, May 05, 2010 9:12 PM.
Alert Triggered  5/5/2010 9:14:57 PM Node TESTNODE has rebooted at Wednesday, May 05, 2010 9:12 PM.

We recieved two TESTNODE Up alert emails - one at 9:13 and one at 9:14. At 10:04 we recieved a TESTNODE rebooted email that stated it rebooted at 9:12. Why did this happen? Other times we get UP alerts for a node that rebooted and nothing else - no down and no reboot alerts. How to prevent this (no down alerts without an up alert - do we then miss valid alerts)?

Second Question - How do we create a alert that can tell the difference between a node that was deliberately rebooted and one that was hard rebooted? With windows nodes, there are event log enteries to tell what happened (not sure even then how we would incorporate that into alert logic). Has anyone build an alert that can tell the difference and show an alert? How would one do this with cisco switches?

Any help is appreciated!
John

NPM 10 RC
NTA 3.6
APM 3.5 RC
NCM 5.5
IPAM 1.6

Find more posts tagged with

Accepted answers

All comments

KLH456

Hate to do a me too... but I am seeing these double reboot messages as well. Not for all nodes though. Now this week, I have received reboot messages from most of my APC UPS devices. When I check their logs there is no reboot noted. Also no traps received indicating a reboot.

We are very puzzled about what is generating the event telling us these devices rebooted when in actuality there is no evidence that they did. This has only begun happening this week. I'm running release 9.5.1 of Orion.

mcbridea

Hi John,

Sometimes this is caused by the trigger and reset timing settings. Do you have any delay set in the trigger? For nodes which can reboot quickly what happens is the node goes down between polling cycles (2 minutes) and then comes up. The down trigger is no longer valid but the rebooted and up conditions are true so you receive those. So the node up and node rebooted without a node down can be interpreted as node down for less than 2 minutes.

For Windows you can use the Windows event forwarder to send the events as syslog and alert on those.

jspanitz

Andy,

We do have delays, those are the 5 seconds and 15 seconds for the trigger and reset. I should have explained that better. Is there any way to ensure that the reboot or up conditions don't fire unless the down condition has fired?

How would one work the syslog option? Would we have to use syslog for all up / down alerts? Or can we somehow link those to the NPM up / downs?

mcbridea

It is possible to have things other than ICMP declare a device down but because the availability enging uses ICMP, it will show as up in the next polling cycle if it is reachable. I would try removing the timers. The reboot condition will fire anytime the condition is true.

ebradford

John,

I'm not sure I fully understand the situation. Have you sometimes gotten the same alert to trigger and actions perform correctly and other times not? Or is one alert never triggering correctly? You are using basic or advanced alerts?

Just a thought. In the Node tree, what is the polling interval under Interface Details? If you are polling every 120 seconds, it is unlikely that you'll get a 5-second response to a node going down.

Mcbridea, correct me if I'm wrong, but don't the Alerts check against the status in the database? That the alert does not go to check the actualy target for the current status, it just refers to the database?

For example: let's say I am supposed to tell my boss when a light blub burns out. But, I can't look at the light bulb, all I can do is look at a piece of paper that my co worker writes on the paper if the light bulb is workng or not. So, every 5 seconds I go look at the piece of paper, and if it says that the light bulb burned out, then I go tell my boss. However, my co worker only goes and looks at the lightbulb every 2 minutes. So, my boss could possibly have to wait 2:05 before knowing the bulb went out. In this situation, I am the alert engine, my co worker is the poller, the piece of paper is the database, lightbulb = not working is the trigger.

Eric

mcbridea

That is true Eric, the alert engine checks the db but the status in the db is dependent on the polling period and last poll.