Network device reboot alerts - multiple alerts from a single site

Hi all, I am new to the forum, but have already made great use of it on many of my Orion installations since upgrading to 8.5, so thanks to everyone who has posted previously, you are all network geniuses...

OK, flattery over, I am looking for an odd, and probably quite bespoke solution. I use NPM on a customer with a large MPLS infrastructure, and each site is monitored individually. I have set up lots of clever advanced alerting with dependencies, and Tiers etc, but what I now want to work on are the device reboot alerts.

Basically, I have a system which auto-logs NPM events into our Incident tracking system, so any false alerts are a pain and require lots of admin to process, so I need to minimise the number of them…

Each of my sites generally have 3-4 devices on them (router, switch, access-points etc), and quite often, power dies at a site, and the devices all go down (sometimes for only a second) but this means I get 3-4 reboot alerts. What I want, is to be able to configure an advanced alerting rule that sends me an email with a title of something like:

"Event: A device on site 'West-London-1' has rebooted".

The event text would tell the support engineers that they should log into Orion and check which devices have rebooted, and check that everything is now operational.

Does anyone have any ideas how to achieve this? I have already created an advanced rule for this, but I get the same alert for each device when the site goes down and up. I am out of ideas...

Help would be much appreciated

Find more posts tagged with

Accepted answers

All comments

grantsewell

I think you might be able to acheive this - but it would be very tedious.

If you wanted alerts from a site, you could always add a custom property with the site name, and create an advanced alert for, say:

IF SITE="West-London-1" AND Node Status=DOWN

But, that would continue to alert you. To suppress additional alerts, you would have to create very detailed suppression rules specific to the Node names and statuses.

Unfortunately I think your kind of stuck. Personally, I'd love to see an ability to do this - I have a similar situation with several small sites, and I get tons of pages to my cell phone every time the power fluxes.

Anyone else have more ideas?

FormerMember

Hi there, Thanks for the reply, it is nice to know someone else is looking into such things too. However, it sounds like you are after something that prevents multiple "down" alerts when a site goes down (i might be wrong here of course) and if you are, this can be achieved taking your "site" property a step further and then assigning a "Tier" custom property to each of your nodes, then create a rule per site with suppression saying that you should not generate any alerts for devices that have a Tier less than that of the gateway router... my problem is, that I like to get notified when Last Boot changes on a node (which might imply an issue with a device, even though service has been resorted) . Some devices can reboot so fast they do not generate a Down alert during normal thresholds so the reboot alert cam be the last chance of identifying a problem... One horrible solution would be to create 3-4 rules per-site, so that there are dependencies set on each device at a site, is one reboots, just send an alert saying that something rebooted on Site-A but suppress if you have already send one for a device closer to the router, or the router itself... any thoughts people?

grantsewell

Right - I was referring to being able to suppress any additional alerts from the same site. I like your idea of using the LastBoot property instead of Node down (I think I just overlooked that in the first post - apologies). I think the only solution might be the horrible one - multiple rules in order to detect additional reboots, and then suppress the further alerts. I'm not sure exactly how to define this feature, but it would be nice to see in future releases. I'm sure it's a common issue.