Monitoring EIGRP Neighbors with custom SQL or SWQL

We have an environment where all routers sharing a particular custom property need to be monitored for a very specific condition.

Each of these routers has two virtual interfaces: Tunnel10 and Tunnel20

Each of these tunnels has a single EIGRP neighbor.

The EIGRP neighbor addresses themselves are DYNAMICALLY assigned, so the actual EIGRP neighbor address itself is not significant to us. Only the presence of an EIGRP neighbor is important.

This output shows a normal operating status:

someroutername#show ip int br | in Tunnel
Tunnel10 10.255.5.59 YES NVRAM up up
Tunnel20 10.255.9.223 YES NVRAM up up
someroutername#show ip eigrp nei
EIGRP-IPv4 Neighbors for AS(65001)
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
1 10.255.5.1 Tu10 11 1d00h 92 1398 0 125120116
0 10.255.9.1 Tu20 10 5d04h 43 1398 0 96867940
someroutername#

The following output shows an ABNORMAL operating status if it persists for more than five minutes:

RT-841-VPA-IL-ELKGROVEVILLAGE#show ip int br | in Tunnel
Tunnel10                   10.255.5.59     YES NVRAM  up                    up      
Tunnel20                   10.255.9.223    YES NVRAM  up                    up      
RT-841-VPA-IL-ELKGROVEVILLAGE#show ip eigrp nei 
EIGRP-IPv4 Neighbors for AS(65001)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
1   10.255.5.1              Tu10                     11 1d00h      92  1398  0  125120116

Note that in the previous output, one of the tunnel neighbors is not working - there is no established neighbor.

I want to create an alert that flags the above condition.

To be specific: If one of the two neighbors is down for more than five minutes, trigger the alert. If that specific condition no longer exists, CLEAR the alert automatically.

I've asked for help on this type of alert before. Syslogs and trap messages don't work for us because they cannot automatically clear themselves and we really don't care if a neighbor disappears for a few seconds, we only care if a neighbor disappears for more than five minutes. We don't care about the actual neighbor IP addresses themselves, only that there IS one.

If neighbors on BOTH tunnels disappear, the router (entire node) goes offline and we have a different alert for that.

Out of the box "routing neighbor" conditions don't work for this set of conditions, many have been tried with the assistance of Solarwinds technical support. They recommend that I come here for attempting to create a custom SQL or SWQL alert.

Can anyone assist?

Find more posts tagged with

EIGRP Alerts

neighbor state

Accepted answers

All comments

cnorborg

So, may be totally off track, but it sounds like you're doing some sort of dynamic neighbor assignment, like DMVPN or something similar. I found it easier to track the interface state of an interface, which is always "up" otherwise, by using the "if-state nhrp" in the tunnel interface config. That way the up/down state of a tunnel matches the nhrp state, making normal alerts usable.

Even if that doesn't fit your needs, you might look further into the "if-state" command, as it allows you to also use the "track" keyword in place of nhrp, which allows you to use the array of "object tracking" methods that Cisco has. ie:

https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5500/sysman/62x/b-system-management-cg-ncs5500-62x/b-system-management-cg-ncs5500-62x_chapter_0100.pdf

But, if none of that works, you'll also need to look at some other things to get what you're trying to get working to fit your needs. SW doesn't poll things like neighbors and such on a very frequent basis, unless your ok with quite a few minutes or hours to go by without knowing if a neighbor is there, you will probably need to up your polling intervals quite a bit on at least those objects. Let me know if the "if-state" stuff is a dead end and I'll try and look into it further...

flackovic

What does the following query return for you? I only spot checked this against one of my devices using EIGRP, but it seemed to return the neighbors that were no longer available when compared with the "Routing Neighbors" widget on the Node Details -> Network tab. Replace line 5 with your actual custom property information. If this returns what you're looking for, you could potentially work this into a custom alert if the polling for routing neighbors is sufficient for your use case.

SELECT n.Caption, n.IPAddress, rn.NeighborIP, rn.ProtocolName, rn.ProtocolStatus, rn.ProtocolStatusDescription, rn.Status
FROM Orion.Routing.Neighbors rn
JOIN Orion.Nodes n ON n.NodeID = rn.NodeID
WHERE rn.ProtocolName = 'Cisco EIGRP'
AND n.CustomProperties.BLAHBLAHBLAH = 'BLAHBLAHBLAH'
AND rn.ProtocolStatusDescription IS NULL

Chris-S

flackovic: That query, run in SWQL, returns the following table (with Caption column removed), with two entries for each individual router, and all the routers currently have a single eigrp neighbor for each tunnel, and there are two tunnels per router, so we get 340 rows returned (I've only included the first ten rows for brevity)

10.0.10.1	10.255.11.1	Cisco EIGRP	0	NULL	NULL
10.0.10.1	10.255.7.1	Cisco EIGRP	0	NULL	NULL
10.0.100.1	10.255.7.1	Cisco EIGRP	0	NULL	NULL
10.0.100.1	10.255.9.1	Cisco EIGRP	0	NULL	NULL
10.0.101.1	10.255.5.1	Cisco EIGRP	0	NULL	NULL
10.0.101.1	10.255.11.1	Cisco EIGRP	0	NULL	NULL
10.0.102.1	10.255.7.1	Cisco EIGRP	0	NULL	NULL
10.0.103.1	10.255.5.1	Cisco EIGRP	0	NULL	NULL
10.0.103.1	10.255.11.1	Cisco EIGRP	0	NULL	NULL
10.0.104.1	10.255.11.1	Cisco EIGRP	0	NULL	NULL

None of our routers currently have a tunnel interface WITHOUT a EIGRP neighbor. (though that is the condition we want to alert on if it happens)

Chris-S

cnorborg: I'm looking into this to see if it works for us - I will respond back later today.

Chris-S

cnorborg: We are using DMVPN however, the entire nhrp setup is far too complex and involved to implement for this one purpose. Suffice to say, we cannot use this as a solution. I do thank you for bringing it to my attention.

We do not need to know right away if a neighbor is missing on a tunnel. We only care if the condition persists. I said five minutes, but we'd likely only care if it was more than an hour. You see, both tunnels provide access to the same resources; they are redundant for each other. If one goes loses a neighbor for bit it doesn't affect anything at all. It just removes a redundancy. If it goes down and stays down, however, we need to investigate because their redundancy is gone. The polling load on our Orion engine isn't high, we could increase the polling rate for neighbors if needed. So.... the "if-state" isn't going to work from us but I certainly welcome any alternative method you might know.

Chris-S

Bump to see if anyone has more ideas...

chad.every

Best I can think of is if you can get SAM/HCO (Not sure what licenses you have), and build an application template that will SSH into the device every 5 minutes, run your show command, and then parse the results. Then an alert of must exist for X time.

The SSH part would have to be done with a script monitor like PowerShell/Python and account for auto-accepting the ssh fingerprint and passing login creds.

bobmarley

Possibly research the MIBS for the device and see if there is one you can use for a Custom Poller for this condition.

chad.every

What i've typically seen with DMVPN is that the OID's are dynamic. So building a custom OID poller isn't feasible.

bobmarley

Thanks! @chad.every That makes it difficult Back to your plan and SSH to get the data.

Step 1. Somehow get the data into the SolarWinds database. Step 2. Create an alert rule or advanced SWQL rule to alert on the condition. Step 2 is easy if you can get past Step 1

adam.beedell

This is a fun one

We want some right joins I think

Do you have an example from the routing.neighbors table of a live issue?

adam.beedell

@Chris-S

whats this do for you?

select distinct(n.caption)
from orion.nodes N
left join
(SELECT n.nodeid
FROM Orion.Routing.Neighbors rn
JOIN Orion.Nodes n ON n.NodeID = rn.NodeID
WHERE rn.ProtocolName = 'Cisco EIGRP') e on e.nodeid =n.NodeID
where e.nodeid is null
and n.vendor like 'cisco'

That aint it, but like, something around there.

We want

a) routers you care about
b) Routers with tunnels

c) Routers with tunnels with EIGRP Neighbours

and you want to alert on the intersection of a subset of A and B but not C or something

-Dark blue bad?

cnorborg

So, I think some of the others have done some of the work for you on the basic queries needed. Here are my thoughts on what direction to go. First, set up a custom property to distinguish what nodes >should< have the pair of neighbors that you mentioned, that way you can query for devices that should have neighbors. Otherwise if a box has no neighbors, it will simply not be there, but with a right join you can pull them in and know that they should have a neighbor. If all your looking at is to see if a box that should have neighbors either has some or doesn't, you can simplify things by not looking at multiple rows of data and trying to parse that somehow, but to simply use a COUNT() to see the number of neighbors they have? So, with how you described it, if the count is 2, you should be good. Count = 1, not so good. Count = 0, you'll want to alert.

Then there is the matter of polling often enough to get it. I wouldn't modify the polling frequency of all the nodes, but try going into "Edit Node" and under "Polling" I think what you need to modify is the "Collect statistics every XX". I'd recommend getting a test node that you can bring neighbors up and down to see how well it works out for you. Waiting for production nodes to go up and down won't work very well..

But, I think this is the general direction to go on this. The hard part might be getting the data to update quickly enough for your needs. If what I suggested doesn't update quickly enough, you might reach out to SW to figure out what adjustment you need to make. Actually, not a bad idea to do it anyway.

HTH?

Chris-S

Hey, cnoborg!

Custom property defining the scope of nodes is already set.
Each router has a Tunnel10 and Tunnel20 interface.
Each tunnel interface should have exactly one EIGRP neighbor.

If Tunnel10 or Tunnel20 are NOT DOWN
AND if either does NOT have an EIGRP neighbor (count=0) for X minutes, generate alert.

The alert will need to clear itself IF the above conditions are no longer met.

As for the minimum minutes the condition would have to be true before generating an alert, I doubt we'd ever need to know if that value was less than 60 minutes.

wesleykparker

You hopefully already figured this out. But the approach proposed by @cnorborg is the way that i would do it. Shouldn't be difficult. A custom SWQL alert for nodes that looks like the following should do the trick. Use the built in alerting features in alert builder to set how long the status must be active to achieve your goals on only alerting after 5 minutes and to clear after alert does not exist.

SELECT Nodes.Uri, Nodes.DisplayName FROM Orion.Nodes AS Nodes -- This line included in the Custom SWQL alert, only copy what's below

INNER JOIN

( SELECT Nodes.Uri, COUNT(Nodes.Uri) AS NeighborCount FROM Orion.Nodes AS Nodes

LEFT JOIN Orion.Routing.Neighbors ON Nodes.NodeID = Neighbors.NodeID

WHERE Neighbors.ProtocolID IN ('16') AND Nodes.CustomProperties.<Customer Property Name> IN ('<Custom Property Value>') -- Replace the two <values> GROUP BY Nodes.Uri ) NeighborCounts ON NeighborCounts.Uri = Nodes.Uri

WHERE NeighborCounts.NeighborCount = 1