15 Replies Latest reply on Jan 25, 2018 11:00 AM by Craig Norborg

    Easy way to monitor DMVPN tunnels!!

    Craig Norborg

      So, I've been experimenting around with different ways to monitor my DMVPN tunnels on remote routers.  If the physical circuit they're running over is still in an "up" state even though there is no Internet connectivity, the state of the DMVPN tunnel tends to stay "up" also, causing monitoring nightmares.   I've been doing a lot of playing with EIGRP status and alerting on that, but the EIGRP alerts are difficult to translate to a "down" state on the node summary or details pages, not to mention the issue with how often EIGRP is polled by default.   So, I finally broke down and decided to look closer and custom MIBs and such to monitor the tunnels with.  While doing so I think I found a much MUCH easier way!!

       

      Was looking for resources on how to monitor DMVPN via SNMP and came across this article:

      DMVPN Tunnel Health Monitoring and Recovery  [Support] - Cisco Systems

       

      I got specifically interested in the part about "interface state control".  Now, this is only useful from the remote routers perspective because what it does is make the state of the interface (ie: whether its up or down) dependent on whether there are any active NHRP sessions for that tunnel.   And all you have to do to configure it is put "if-state nhrp" on the tunnel interface, super easy!

       

      Now, this might not be perfect, but its pretty good.   Why not perfect you might ask?

       

      Let's say you have dual DMVPN hubs and the connection to one of the hubs isn't working on a given tunnel due to a routing misconfiguration or something.  Since there would be one active NHRP session on the tunnel interface, this wouldn't find it.

       

      This also does nothing for monitoring the tunnels from the head-end perspective, if for some reason monitoring the remote routers wasn't how you wanted to do this.  Once again, not useful.

       

      You might also say if all your DMVPN tunnels weren't functioning, that this wouldn't help, but in that case the whole node should be down so I think we can discount that!

       

      Not much to lose, and pretty easy to configure up, give it a try and let me know what you think!

        • Re: Easy way to monitor DMVPN tunnels!!
          Craig Norborg

          And to make it easy on everyone who wants to use it, here is a NCM Compliance rule that will help you remediate those nodes that don't have this on your DMVPN tunnel interfaces...    Our DMVPN tunnels are all either Tunnel11 or Tunnel21, which is why the "Config Block Start" regular expression is how it is, adjust to your own environment.

           

          if-state nhrp.png

           

          One question for those brave enough to implement this rule, when remediating all nodes, does your Orion seem to lock up and only do a limited # at a time?   Mine seems to only do about 30 or so before I have to cycle my server and start things over.   Still quicker by far than doing it manually, but a bit of a pain ...

          2 of 2 people found this helpful
          • Re: Easy way to monitor DMVPN tunnels!!
            temark

            We run BGP over our DMVPN tunnels, and monitor that.  I would think you could monitor EIGRP status the same as BGP?

            • Re: Easy way to monitor DMVPN tunnels!!
              Craig Norborg

              Yes, as I stated I am monitoring EIGRP, but it doesn't quite give me what I want.  This method where it brings the tunnel interface down when NHRP doesn't establish, is not only much easier, but gives me better results for my dashboards and such.

              • Re: Easy way to monitor DMVPN tunnels!!
                Craig Norborg

                Correct, and how it may or may not differ depends on your environment.   If you have WAN interfaces on your DMVPN router that reflect the actual state of the link, such as a T1 or T3, when the T1 is down the associated DMVPN tunnels would be down.   However, if the majority of your network is like mine, where we have our router connected into a cable modem or some other type of high speed connection that hands you an ethernet connection, the ethernet link between your router and cable modem is almost always up as long as the cablemodem has power.   In this case the state of the DMVPN tunnel will remain up even if your router can't communicate over that link.

                 

                This approach solves that.  It makes the tunnel interface state be up or down depending on whether there are active NHRP neighbors on the link.   So, if you have a cable-modem hanging off of Gi0/2 on a router with a DMVPN tunnel on it, you would still see Gi0/2 in the up/up state, but the DMVPN tunnel would go down when NHRP went completely inactive on the link, so you would see the down interface in Orion!

                 

                This also depends on your remote sites having redundant links.   If you have only a single link, you can just as easily track the reach-ability of the node itself and assume if the node is down that its WAN link is also.

                 

                I'm sure there are other caveats to this approach, but have yet to think of them!  Overall though, for me and I'm sure others, this will save a lot of time!

                2 of 2 people found this helpful
                • Re: Easy way to monitor DMVPN tunnels!!
                  daks123

                  Hi, So I want to monitor the tunnel status (up/down) and dynamic protocol (Ospf) running in for dual DMVPN set up. What's the best way to achieve this in Solarwinds ?

                    • Re: Easy way to monitor DMVPN tunnels!!
                      temark

                      You can set alerts for the tunnel interfaces, as well as the tunnel-source interface.  You should be able to monitor the status of OSPF over those tunnels the same as BGP or other routing protocols.

                        • Re: Easy way to monitor DMVPN tunnels!!
                          daks123

                          Cool, So through SNMP trap can we monitor this and if yes, whats the best mibs available for this requirement ?

                            • Re: Easy way to monitor DMVPN tunnels!!
                              temark

                              I would build an alert like this:

                               

                                • Re: Easy way to monitor DMVPN tunnels!!
                                  daks123

                                  Hi, I guess your condition will be applicable to all WAN router Ospf related issues. I have a query here, Let's say we have a spoke router which is connecting to a hub router. As per the trigger conditions set by you , if we set the same in Hub router it will be working but the same set in spoke router won't do much good for us because from spoke router we have 2 OSPF running, one connecting to hub through tunnel and the other OSPF running between another secondary spoke router. We need OSPF alert only for tunnel related communication. Is this possible ?

                                    • Re: Easy way to monitor DMVPN tunnels!!
                                      temark

                                      The "WAN Router" line is a custom property I made for our environment.  We build different alerts for different equipment roles, I probably shouldn't have put that in the screen shot.
                                      To get OSPF alerts only over tunnels....  Why not just do the alerting from the spoke routers?  Alerting for the same thing on the hub router would just give you a redundant alert anyway (if I understand your scenario.)

                                      We do lots of BGP over DMVPN, and only monitor/alert for the BGP from the endpoints.  We purposefully exclude the DMVPN Hub routers from those alerts.

                                       

                                      Here's our actual BGP alert trigger conditions with DMVPN hub router names and IP's redacted, as well as a certain segment of the business that we excluded where we use a more specific alert:

                                      You can model off of that, and change routing protocol to OSPF instead of BGP.  You can of course customize it to fit your environment.

                                        • Re: Easy way to monitor DMVPN tunnels!!
                                          daks123

                                          Hi, Your point is correct but just asking you if this logic works When you say that you purposefully neglect alerts/monitor from hub router as it duplicates the alerts , Why not just put the monitoring/alerting only in hub router ? Because, I believe all the spoke will report to hub in such a way if we monitor Hub router(Ospf) communication will notify us in case of any peer down for a particular spoke to hub . Also in our set up we have lots of spokes for which it will be difficult for us to frame the scripts for each spoke. If there is any better way I am happy to hear it please.

                                           

                                          As said, we have a dual DMVPN set up, my focus is to set up tunnel monitoring but Solarwinds technical team confirmed that there is no way to get hubs and spoke information and it is in development mode. So we thought even if we monitor the protocol running on tunnel that would serve the purpose . Ideally if there is a tunnel down its protocol also gets impacted and thereby based on protocol alert ,we can get to know the tunnel info.

                                            • Re: Easy way to monitor DMVPN tunnels!!
                                              temark

                                              We have a lot of DMVPN tunnels coming back to out two hubs.  These are all backup links to primary MPLS.  With that number of tunnels, and most of them being over consumer internet connections, we get a few drops a day.  We like getting the alert for the end sites rather than on the hub router, as we then know immediately which site is affected, without having to go track down which BGP session is down on the hub router.

                                               

                                              You can of course, set it up however works best for you, this is what we found works best for us.

                                • Re: Easy way to monitor DMVPN tunnels!!
                                  michel.salloum

                                  Hi Craig, everybody.

                                   

                                  I run BGP over our DMVPN tunnel , but when i created a new rule in NCM settings and tired to test it against  the router , i've got this  msg :

                                   

                                  knowing that the tunnel interface is tunnel 1  in my config.

                                   

                                  can you advise please.

                                   

                                  Thank you in advance.

                                   

                                  • Re: Easy way to monitor DMVPN tunnels!!
                                    Craig Norborg

                                    That rule would check to see if you have this configured on your DMVPN tunnels.  If it's violated, that means its not.   If you run the remediation script on those interfaces it should put the configuration in them, or you can do it manually.

                                     

                                    he compliance rule just checks to make sure your interface is configured for it.  Once the interface is configured properly the interface will go up or down based on the NHRP state of the interface, so it will be just like any other interface.

                                     

                                    Does that make sense?