12 Replies Latest reply on May 15, 2010 10:45 PM by bshopp

    Automatically Add/Remove Interfaces on Node From Polling (Attempting Clever SQL and Failing)

    GaryP

      Okay...allow me to provide some background first: we've got a large number of Cisco switches and routers on our network (827) and there wasn't enough thought put into what interfaces should be monitored as they were added to NPM and it's now gotten a little out of hand.

      We're working on a classic three-layer model of core, distribution and edge switches and, broadly speaking we'd like to monitor all interfaces on all core and distribution switches, regardless of status (Up, Down, Administratively Down) but not monitor any of the 'edge ports' on the edge switches, just CPU, Memory, etc and the gig uplinks to the distribution layer. There's also a fourth layer, edge switches in the server rooms, where we'd still like to monitor all the ports, all the time.

      I've put in some preparatory work and created a custom node property called 'Network_Layer' with values as follows for all our network equipment:

      10 = Core

      20 = Distribution

      30 = Static Edge (i.e. server switches, all ports monitored)

      40 = Dynamic Edge (i.e. 'user' edge switch, only uplinks monitored)

      99 = Other (devices that are uncategorised and we don't want to automate port management for)

      I've also added a true/false custom interface property called 'No_Auto_Manage' that will be used to override the 'Network_Layer' for specific interfaces on a device (i.e. a server sitting on a switch with Network_Layer = 40).

      Now is where I come undone. I was hoping to find all interfaces for a node listed in the database with a simple monitored or unmonitored switch and issue SQL queries to turn monitoring on or off based on the 'Network_Layer' for the device (with 'No_Auto_Manage' on an interface allowing an override).

      Alas, only the monitored interfaces are listed. It seems that the 'List Resources' procedure actually SNMP walks the device in question and adds and removes interfaces from the database based on the selections you make.

      So, can anyone think of a way I can automate the 'List Resources' procedure somehow? I've got more than 820 devices I want to set the interfaces up on and doing it manually for every device is going to be very labour intensive.

        • Re: Automatically Add/Remove Interfaces on Node From Polling (Attempting Clever SQL and Failing)
          ecklerwr1

          I think you are right that only monitored interfaces are in the database.  It would seem the only way I can think of you to do you've already firgured out in theory.  You'll have to MIB walk the Interfaces MIB to find all the present interfaces.  All of these interaces have an index which can be confusing because, from my experience, each time a new module inserted in say a Cisco router... the interfaces are renumbered ie. their index might be different now.  I believe this is the real reason SW does the rediscovery process... they look to see if the currently monitored interfaces have changed due to this reindexing that occurrs when something changes with the hardware so they still monitor the original selected interfaces.

            • Re: Automatically Add/Remove Interfaces on Node From Polling (Attempting Clever SQL and Failing)
              GaryP


              All of these interaces have an index which can be confusing because, from my experience, each time a new module inserted in say a Cisco router... the interfaces are renumbered ie. their index might be different now.

               



              Try this magic sauce out on your routers , I believe it'll sort you out in this respect (I know this works on our 6500s, ymmv):

                 Router(config) # snmp-server ifindex persist

              this will operate on the whole system. If, however, you only want to make certain interfaces 'sticky' you can issue:

                 Router(config-if) # snmp ifindex persist

              Hope that helps!

            • Re: Automatically Add/Remove Interfaces on Node From Polling (Attempting Clever SQL and Failing)
              darryld

              could you run a discovery covering the IP range of the equipment involved and during the import phase select the relevant device types and interface types that you  are interested in.

              This would add all the interfaces into the database. You could then set all those which you do not want stats for to the "Unmonitored" state

              1 of 1 people found this helpful
                • Re: Automatically Add/Remove Interfaces on Node From Polling (Attempting Clever SQL and Failing)
                  GaryP


                  could you run a discovery covering the IP range of the equipment involved and during the import phase select the relevant device types and interface types that you  are interested in.

                  This would add all the interfaces into the database. You could then set all those which you do not want stats for to the "Unmonitored" state

                   



                  That's one option but, as I've already got all the nodes I want to monitor in the system, wouldn't we loose the historical data from them if I re-imported?

                    • Re: Automatically Add/Remove Interfaces on Node From Polling (Attempting Clever SQL and Failing)
                      darryld

                      No

                      The import will identify that the node already exists a skip adding the node and move on to the interfaces. the node history will be unaffected

                        • Re: Automatically Add/Remove Interfaces on Node From Polling (Attempting Clever SQL and Failing)
                          GaryP

                          Ah, Darryl, that makes sense!

                          If I re-import my nodes a netblock at a time I can import them with all interfaces monitored then strip out the unwanted ones with some SQL and not overload the pollers while I'm doing it.

                          I'll report back how I get on later...

                            • Re: Automatically Add/Remove Interfaces on Node From Polling (Attempting Clever SQL and Failing)
                              GaryP

                              Bad news, I was wondering why my poller is now showing a significantly higher number of interfaces being monitored now, much more than expected. All the nodes that have had interfaces added since the re-import have actually been duplicated, rather than simply adding the interface to the existing node. This is really causing me problems with polling completion now. Fortunately I only re-imported for a small section of my network but I'm still looking at 85 switches to go through and remove now...

                               

                              Update...

                              Note to self for future work: do not tick the "add even if node exists on other pollers' box. All duplicate switches removed now and interfaces reimported.

                                • Re: Automatically Add/Remove Interfaces on Node From Polling (Attempting Clever SQL and Failing)
                                  GaryP
                                  Well, I worked my way through this one sectin of my network at a time and I'm now happy with the state of monitoring on the network.
                                  A SQL query was needed that would reliably select and delete interfaces with the following criteria:
                                  - fast ethernet ports
                                  - without the no_auto_manage flag set
                                  - on nodes with a vendor of 'cisco'
                                  - with a given vtp_domain
                                  - with a network_layer of 40
                                  A couple of wrinkles popped up, though: 2960-G24 and 2960-G48 switches only have gigabit interfaces so some logic was added that would leave the last two interfaces (23/24 or 47/48) monitored as these would likely be the uplinks. (Stacks will present another problem when they're used for 'user' edge situations but I'll deal with that when it arises).
                                  Here's the secret sauce:

                                  delete interfaces
                                  from interfaces left join nodes
                                  on nodes.nodeid = interfaces.nodeid
                                  where nodes.vendor = 'cisco' and nodes.vtp_domain = 'east park' and network_layer = 40 and interfaces.no_auto_manage = 0 and interfaces.interfacename like '%ethernet%'
                                  and (
                                  interfaces.interfacename like 'fastethernet%'
                                  or
                                  (interfaces.interfacename not in ('gigabitethernet0/47', 'gigabitethernet0/48') and nodes.machinetype = 'Cisco Catalyst 2960-G48')
                                  or
                                  (interfaces.interfacename not in ('gigabitethernet0/23', 'gigabitethernet0/24') and nodes.machinetype = 'Cisco Catalyst 2960-G24')
                                  )

                                  It's worth noting that there's a lot of data in other tables within the database relating to each individual interface, particularly historical data. There's no need to worry about tidying up after yourself, though, as an internal nightly database maintenance job clears out any orphaned entries.
                                  We're also starting to see more of Cisco's stacking switches appear and they're going to present another challenge in the naming of interfaces within the stack but I'll cross that bridge when I come to it.
                                  Thanks to everyone for all their help with this one. It was a real stinker and it's all sorted now.
                                  1 of 1 people found this helpful
                        • Re: Automatically Add/Remove Interfaces on Node From Polling (Attempting Clever SQL and Failing)

                          I have a similar network architecture but approached the problem in a different way. I chose to monitor all switch ports on all devices (just to have traffic stats), however I marked all the ports on the user switches as 'unpluggable'. It is fairly simple with a little SQL to mark all the interfaces of a given node unplugabble. After that, I just manually fixed the uplinks to be monitored full time.

                          Finally, in my advanced alerts for interface down, I test for 'unplugged' as a status and dont alert if i see that.

                            • Re: Automatically Add/Remove Interfaces on Node From Polling (Attempting Clever SQL and Failing)
                              GaryP


                              I have a similar network architecture but approached the problem in a different way. I chose to monitor all switch ports on all devices (just to have traffic stats), however I marked all the ports on the user switches as 'unpluggable'. It is fairly simple with a little SQL to mark all the interfaces of a given node unplugabble. After that, I just manually fixed the uplinks to be monitored full time.

                              Finally, in my advanced alerts for interface down, I test for 'unplugged' as a status and dont alert if i see that.

                               



                              I considered that but one of the reasons I want to completely have the interfaces out of the system is the sheer volume of them. I've got a better idea of the numbers since I labelled the devices as described in my OP: just doing a count of 24- and 48-port switches in my 'layer 40' I've got 17,592 interfaces that I don't want to monitor. That's more than two pollers' worth! I can't afford to have that amount of interfaces being polled for no reason.

                              Thinking about this a little more overnight I realised that it's actually much easier for me to delete interfaces from the database if they're already there (i.e. the node is added to the system with all interfaces checked, as Daryl suggested) than add them in (as this requires external snmpwalk functionality). It'd be great to have all the interfaces in there but unmonitored ones 'hibernated' but that's obviously not the way Solarwinds have built it.