NPM 10.5 Routing Protocol Monitoring - scalability

Hi there...

We just upgraded to 10.5 and as soon as we brought the system back online it "crawled" for performance very badly. One of the new features we were quite eager to see in action was routing protocol monitoring. We quickly learned that it was the reason for all of our sudden grief.

It would appear that when Solarwinds tested this feature that they didn't consider the size of typical routing tables in a Service Provider environment. My long standing complaint about SW has always been that they think with their "Enterprise" hat on and forgot about the Service Provider customers they have (for now).

So I rolled a ticket with support to find out how to disable routing protocol monitoring and was told that I now have to go into each node and disable this support. As such, this will take us potentially several days to complete which is a major disappointment. We wanted to globally disable these features and have been told it's not possible.

My feedback is to consider service providers when doing your testing. Also, your tech support people should be able to open feature requests instead of having the customer do it. Finally, for features that could have a dramatic effect on an existing installation I would suggest it become an option to enable during the upgrade vs forced upon us - especially if there's no simple way to disable that feature.

Thank you for listening,

Paul

Find more posts tagged with

routing

npm

protocol

monitoring

Status: None

Comments

jhay317

This was brought up during the beta and I thought the solution was to limit the amount of routes polled to a few thousand as we saw this issue in the Service Provider environment too. Are you seeing SW poll the entire table and then limit the amount displayed or limit what it is polling at all? We are looking at upgrading our production server but some of my concern was around if they enabled this polling by default on all nodes and it looks like some of my initials fears are justified.

pstewart726

Thanks - I'm not sure if it's pulling the entire table or not but the entire system crawled for hours until we started finding ways to turn off the routing table polling. It only displays 1000 entries but again not sure if it's actually trying to store/poll the entire tables. We are still going through each router manually turning off everything except OSPF neighbour monitoring at this point and things are starting to get faster. We may re-enable BGP neighbour monitoring but want to be careful there as some of our routers have 700-800 BGP neighbours and not 100% confident in what impact that will have.

It really bothers me though that this feature was turned on by default - and now we have to spend a lot of man-hours running around doing a "list resources" on each device, wait for it to finish, and then un-click.

jhay317

Thanks for the information. We tested polling about 200 BGP neighbors on one router and about 30 on couple another routers in the Beta and RC and didn't see any issues from that but that is only three routers.

I agree that shouldn't be needed. We noticed something similar with the topology polling too. SW seems to assume that when they come out with a new feature we want everything on the network now polled for that new feature instead of a more phased approach to rolling it out. I wonder if somewhere in the database there is a flag that could be manually unset to turn off that polling.

It would also be nice if they supported ISIS polling.

pstewart726

Yup - couldn't agree more. We went into panic mode and didn't look at the database (plus quite honestly we're not database people) but there should be a field that could be adjusted.

ISIS would be nice - we don't use it today but it is common in SP environment contrary to what some folks think

jhay317

It looks like at least in the RC they were only putting the limited amount of routes into the routing table. Have you tried reducing the max routing table rows like in this article?

SolarWinds Knowledge Base :: Fixing routing table resources with incomplete data

I realize this is a complete workaround but might help for a little more immediate relief.

pstewart726

Thanks – yes, it defaulted to 1000 which is fine but I think it's the number of routers dumping 1000 routes into the system. It's definitely a very cool feature – just too bad we couldn't use it for anything of scale… hopefully this can be addressed in upcoming releases, although quite honestly unless Solarwinds is going to load their own routing daemon into the system I don't think it'll scale well using just SNMP

enid.jimenez

Thanks for the feedback. NPM should not be polling or saving the entire routing table, but only the first 1000 entries. Could you please let me know the case number opened with SWI support? I would like one for someone to follow up with you.

FormerMember

"...for features that could have a dramatic effect on an existing installation I would suggest it become an option to enable during the upgrade vs forced upon us - especially if there's no simple way to disable that feature."

I hear you. Could use a vote here.