Automatically Add/Remove Interfaces on Node From Polling (Attempting Clever SQL and Failing)

Okay...allow me to provide some background first: we've got a large number of Cisco switches and routers on our network (827) and there wasn't enough thought put into what interfaces should be monitored as they were added to NPM and it's now gotten a little out of hand.

We're working on a classic three-layer model of core, distribution and edge switches and, broadly speaking we'd like to monitor all interfaces on all core and distribution switches, regardless of status (Up, Down, Administratively Down) but not monitor any of the 'edge ports' on the edge switches, just CPU, Memory, etc and the gig uplinks to the distribution layer. There's also a fourth layer, edge switches in the server rooms, where we'd still like to monitor all the ports, all the time.

I've put in some preparatory work and created a custom node property called 'Network_Layer' with values as follows for all our network equipment:

10 = Core

20 = Distribution

30 = Static Edge (i.e. server switches, all ports monitored)

40 = Dynamic Edge (i.e. 'user' edge switch, only uplinks monitored)

99 = Other (devices that are uncategorised and we don't want to automate port management for)

I've also added a true/false custom interface property called 'No_Auto_Manage' that will be used to override the 'Network_Layer' for specific interfaces on a device (i.e. a server sitting on a switch with Network_Layer = 40).

Now is where I come undone. I was hoping to find all interfaces for a node listed in the database with a simple monitored or unmonitored switch and issue SQL queries to turn monitoring on or off based on the 'Network_Layer' for the device (with 'No_Auto_Manage' on an interface allowing an override).

Alas, only the monitored interfaces are listed. It seems that the 'List Resources' procedure actually SNMP walks the device in question and adds and removes interfaces from the database based on the selections you make.

So, can anyone think of a way I can automate the 'List Resources' procedure somehow? I've got more than 820 devices I want to set the interfaces up on and doing it manually for every device is going to be very labour intensive.

Find more posts tagged with

Accepted answers

bshopp

Not sure if these will help, but check them out, might give you some ideas here and here

All comments

ecklerwr1

I think you are right that only monitored interfaces are in the database. It would seem the only way I can think of you to do you've already firgured out in theory. You'll have to MIB walk the Interfaces MIB to find all the present interfaces. All of these interaces have an index which can be confusing because, from my experience, each time a new module inserted in say a Cisco router... the interfaces are renumbered ie. their index might be different now. I believe this is the real reason SW does the rediscovery process... they look to see if the currently monitored interfaces have changed due to this reindexing that occurrs when something changes with the hardware so they still monitor the original selected interfaces.

darryld

could you run a discovery covering the IP range of the equipment involved and during the import phase select the relevant device types and interface types that you are interested in.

This would add all the interfaces into the database. You could then set all those which you do not want stats for to the "Unmonitored" state

matthewbsimon

I have a similar network architecture but approached the problem in a different way. I chose to monitor all switch ports on all devices (just to have traffic stats), however I marked all the ports on the user switches as 'unpluggable'. It is fairly simple with a little SQL to mark all the interfaces of a given node unplugabble. After that, I just manually fixed the uplinks to be monitored full time.

Finally, in my advanced alerts for interface down, I test for 'unplugged' as a status and dont alert if i see that.

GaryP

could you run a discovery covering the IP range of the equipment involved and during the import phase select the relevant device types and interface types that you are interested in.
This would add all the interfaces into the database. You could then set all those which you do not want stats for to the "Unmonitored" state

That's one option but, as I've already got all the nodes I want to monitor in the system, wouldn't we loose the historical data from them if I re-imported?

GaryP

All of these interaces have an index which can be confusing because, from my experience, each time a new module inserted in say a Cisco router... the interfaces are renumbered ie. their index might be different now.

Try this magic sauce out on your routers , I believe it'll sort you out in this respect (I know this works on our 6500s, ymmv):

Router(config) # snmp-server ifindex persist

this will operate on the whole system. If, however, you only want to make certain interfaces 'sticky' you can issue:

Router(config-if) # snmp ifindex persist

Hope that helps!

GaryP

I have a similar network architecture but approached the problem in a different way. I chose to monitor all switch ports on all devices (just to have traffic stats), however I marked all the ports on the user switches as 'unpluggable'. It is fairly simple with a little SQL to mark all the interfaces of a given node unplugabble. After that, I just manually fixed the uplinks to be monitored full time.
Finally, in my advanced alerts for interface down, I test for 'unplugged' as a status and dont alert if i see that.

I considered that but one of the reasons I want to completely have the interfaces out of the system is the sheer volume of them. I've got a better idea of the numbers since I labelled the devices as described in my OP: just doing a count of 24- and 48-port switches in my 'layer 40' I've got 17,592 interfaces that I don't want to monitor. That's more than two pollers' worth! I can't afford to have that amount of interfaces being polled for no reason.

Thinking about this a little more overnight I realised that it's actually much easier for me to delete interfaces from the database if they're already there (i.e. the node is added to the system with all interfaces checked, as Daryl suggested) than add them in (as this requires external snmpwalk functionality). It'd be great to have all the interfaces in there but unmonitored ones 'hibernated' but that's obviously not the way Solarwinds have built it.

darryld

The import will identify that the node already exists a skip adding the node and move on to the interfaces. the node history will be unaffected

GaryP

Ah, Darryl, that makes sense!

If I re-import my nodes a netblock at a time I can import them with all interfaces monitored then strip out the unwanted ones with some SQL and not overload the pollers while I'm doing it.

I'll report back how I get on later...

bshopp

Not sure if these will help, but check them out, might give you some ideas here and here