Complex Dependency Setup Question

I have some remote sites where the dependency is not auto calculated. These sites don't have direct physical links, but have multiple routes in a routing protocol to them. The remote sites are just one router, but with 2 routes to one upstream router, and 2 routes to another upstream router.

The other day, the remote site router went down, possibly due to storms. SolarWinds kicked off 5 alerts. One for a node down, and 4 from the upstream routers for routing neighbor losses. The alerts are valid, but our support teams would like this incident down to one alert, or more accurately one BMC Remedy ticket. At other times if only a route goes down, we still need that alert.

I am trying to think through the dependencies here. The remote site router is a child of 2 different upstream routers, each with 2 route paths down to the remote site. Maybe it is just me because it is Monday, part 3, but I am not sure how to build this one, or even if it can be done without breaking something else.

Find more posts tagged with

parent child alert dependency

Accepted answers

jkrenzien

This is one way you could setup the alert trigger. The routing neighbor IP is a hub IP. I separate my hubs into different alerts so I can easily identify in the emails which connection type failed (MPLS/ISP). This would also only apply to my branch office.

Neighbor Alert.JPG

All comments

jhandberg

bump

jhandberg

bump

jkrenzien

Alright, first a few a few questions. Is your topology for that part of the network hub and spoke or is it mesh? How are your neighbor alerts currently built?

Now if it is basically hub and spoke I would suggest possible modifying or create a new alert for your network neighbor alerts. i have a similar situation using DMVPN I have just used custom properties to identify devices that are in a data center, remote site, are core devices, WAN devices, etc. Using that identification I have already built I then only alert on my remote sites losing one of the neighbors if is IP is one of my DMVPN hubs (cores). Then the basic fact that the node is down will eliminate the possibility of SolarWinds seeing all of those neighbors down. But this is all assuming your network is somewhat hub and spoke. Either way please let me know and I can post more details of how I accomplished this and help you figure out how to do the same.

cnorborg

Hmm... I have a similar setup. Dual DMVPN Hub routers with 2 DMVPN networks on each hub (Primary / Secondary). Each remote router has two internet links, a primary and a secondary. The Primary link from the remote router, connects to both DMVPN hub1 and hub2 on their primary tunnels, the same with secondary on the secondary links. So, each remote router has 4 routes installed in their routing tables.

Auto-dependency is working fine on both the hub routers and the remote routers. I can see the relationships built for them. We are monitoring the routing protocols on the remote routers and the tunnel interfaces, and the tunnels have CDP neighbor turned on.

That being said, I don't see this as a dependency problem with the alerts. A dependency problem would be if your remote router goes down, it should suppress the switches at that site from alerting, and I'm guessing any other alerts that might be configured for those switches or that router. The problem here is that the remote router is going down and you want to suppress alerts on the hub routers, which are in >front< of the device that went down. Correct? This is the exact opposite what the auto-dependency routines do I believe.

So the question is, how to do it? Or, do you want to do it? I've noticed that, in my case at least, the DMVPN tunnel interface doesn't go down unless the interface goes down. Since we are doing a lot of cablemodems and such where my router only see's the status of the link between my router and the cablemodem, not the status of the WAN link itself, I almost never see the tunnel go down!

So I'm experimenting with EIGRP routing alerts to figure out what's happening. I leveraged some alerts by brian.trotter@polypore.net, shown in the link below, to create a resource off those alerts. In that resource if I see two alerts on the same router for the same interface (ie: tunnelXX), I'm pretty sure that link is down.

Need assistance with NPM alert for EIGRP Routing Neighbors

Guessing your in a somewhat similar situation and like me are trying to think of a better way to do it?

jhandberg

This sounds similar to my troublesome connections. In this case, I am looking at remote sites that have single DSL connections to the internet. There are VPN tunnels back to 2 upstream aggregation routers, and multiple routes through each VPN tunnel going to the upstream routers. I believe the multiple routes to each upstream router are for supporting multiple VLANs/IP Subnets based on what I am seeing. All this is visible in the Routing Neighbors tables. These routes are all Cisco EIGRP.

Our networking group wants to know when routing neighbors go down. Various parts of the WAN use BGP, OSPF, and Cisco EIGRP. Fine, we turned on the canned alert for that. But the service desk said that the number of alerts that occur when something goes down is unacceptable. I often see a remote site completely go down, some have been power outages. This triggers alerts for the node and all route neighbor losses.

I have one remote site where all the routes to one of the upstream routers are down, but the site isn't down because the routes to the other upstream router are fine. In this case, I would think we want to know about the redundancy loss.

It is worth noting that currently auto-dependency doesn't work across the tunnels. These are not direct physical connections. These connections or routes are not being picked up when trying to map these with Network Atlas ConnectNow, or with Network Topology Mapper. But I can trace all the neighbor IP Addresses between the routers and their Routing Neighbor Tables.

cnorborg I think I read the post you mentioned once before, but will review it again. Maybe more has been added since I saw it last.

In reference to jkrenzien's question, much of the WAN network is spoke and hub, but multiple hubs tied together. We cover a large geographical area here. In this post, I was more looking at the remote sites that have DSL or cable connections back to these particular upstream aggregation routers.

jkrenzien

Neighbor Alert.JPG

jhandberg

I can see how that would work for checking redundancy loss from the branch site. An I already know that if the node goes down, alerts don't fire for the routes coming out of the down node, although they do from the upstream routers.

One complication. Each of the many routes have different IPs assigned to it on the remote router. For example, I have a remote site router with a management IP, the one in SolarWinds, and it also has 6 additional IPs for each of the routes. The upstream aggregation routers have different IP Addresses associated with each VRF in addition to their management IP Address. That is 25-35 IP Addresses on the aggregation routers.

So while I can see this would work, I think I would be customizing multiple alerts depending on what VRFs the remote site is supporting. And that is only for the remote sites. It doesn't squelch all the extra alerts coming out of the aggregation routers when they see the routes go down.