Going global with NPM - your best practice recommendations/experiences?

rharland2012

This week some of our European colleagues came for a visit to collaborate, streamline and standardize design, and generally compare notes about how we can best do things as a global company moving forward (I'm part of a group responsible for the Americas).

We stood up our NPM/SAM/NTA installation some months ago for monitoring and managing the Americas network and our European peers were very excited about the Solarwinds products.

My task now is to discover the best method - hopefully from both a price and performance standpoint - of extending the visibility of Orion into EMEA for our business.

General assumptions I have at this point are the following:

1. If we place an additional polling engine in Europe for our current installation it would still have to communicate data to our SQL box here (not a great idea over a finite WAN link), so that's probably not best.

2. The easiest way to get a single pane of glass for all global IT staff would be to implement EOC, while standing up an separate Orion server and SQL backend in Europe or Asia (or both).

I'm not sure these assumptions are correct - but that's why I'm reaching out to the community.

We have about 1600 interfaces here in the Americas presently, with perhaps half of that being added in EMEA.

Thanks for any information (and especially experience) that you can provide.

Find more posts tagged with

eoc

npm

Accepted answers

All comments

branfarm

Hi there,

I currently monitor 5 sites across the globe with my NPM installation -- 3 in North America, 1 in EMEA, and 1 in Asia, with almost 500 nodes and 1800 interfaces spread throughout. EMEA and Asia represent about 20% of the nodes, and probably about 30% of the interfaces. I've found that Orion performs just fine from a central installation, but I suppose a lot will depend on what kind of polling frequencies you want to have. I use the default 2 min polling intervals (for response time and status) and I have 5 min intervals for interface statistics, 10 mins for nodes, and 30 mins for volumes. This has been sufficient for our environment, and we have users in each region who view the main NPM interface remotely without issue.

I don't have the most nodes or interfaces out there, so I don't know how quickly you run into issues as your count increases, but in my experience, you can get away with running a centralized install to monitor global operations.

Hope that helps,

Brandon

rharland2012

Brandon,

thanks for the info! What are the link BWs between your Americas/EMEA/Asia locations?

branfarm

Each site is connected to an MPLS provider, so the bandwidth to each site is really bandwidth into the MPLS cloud. Each NA site has 100Mbps, and both my EMEA and Asia site have 40Mbps.

rharland2012

Brandon, thank you. Have you noticed what kind of bandwidth the Solarwinds-specific traffic generates on these links?

rjnicholson

I would give you a word of caution if your sites aren't connected through MPLS. MPLS takes out the "what if" on Network Latency and what path you will travel each time as opposed to the open INET and whichever BGP route shows to be more viable at that time.

I like to think of MPLS as a protocol that rides the boundaries of being a Layer2 and Layer3 protocol.. It sits nicely in between both. Using labels to route traffic instead of Long Address routing. MPLS is great for Layer 2 connectivity from Site to Site branches across large geogrphical locations. With MPLS you can direct traffic from Site to Site to ride dierctly on your providers core infrastructure, and never leaving out to the INET to get to the next hop. Because of this QoS and Traffic Shaping can be greatly controlled from the 1st hop to the last. This allows for you to properly poll sites across the globe using a central installation if you so choose.

The company I work at is Global and we manage all of our sites 40+ from a Central installation in Dallas, TX. We manage over 2000 nodes and 10,000 total elements on NPM, and monitor over 200 applications with SAM. We also use NCM to back up all of our Cisco and Riverbed equipment (390+ nodes).

We have sites in the UK, UAE, Singapore, and Japan as well. All reporting back to Dallas Texas. The only thing I had to adjust was expected latency coming from overseas. On our MPLS links in Singapore and Japan I see a range of 300-400ms latency coming to and from the site. If you adjust the Orion thresholds accordingly and have proper QoS and Traffic Shapping set up you won't see any issue provided your provider has no issues on their core. This is even nicer if it's your own core infrasturcture and you can really watch what you are pushing across each core router at each POP.

rharland2012

Thank you, Richard. We use MPLS between our sites as well. It's good to hear of positive experiences!

rjnicholson

No problem! MPLS is the way to go, and if you have a decent provider I wouldn't see you having an issue.

Let me put some more ease at mind when I tell you all our MPLS sites are either on a full T1 (1.5 meg) and a back up DSL 1 meg connection using HSRP for HA failover. Some sites have DSL only and we haven't had a single issue with polling/missing pols, or SolarWinds using to much bandwidth and affecting production. I hardly register any traffic to and from sites during non business hours when it's only SW polling the sites.

I would say deploy and start mocking up the scenario so you can see what kind of metrics and statistics you start seeing. Once you have a good burn in you can start to dump everything into your build.

The other option you mentioned of pollers in separate locations I wouldn't recommend for the same reason you stated of wrting back to a central SQL even if they are on MPLS links. If you go this route I would install NPM and a SQL database on each build out. Then use the EOC to manage them at a central point.

bobby.alvarez

Richard, you are only using only one Orion NPM server for all 10,000 elements? At some point I was told by Solarwinds that 6,000 elements was a limit, per server. We are also considering consolidating our global monitoring solutions to just Solarwinds and was thinking I would have to add another server in the EMEA.

rjnicholson

One NPM and 1 Additional poller. I load ballance them by Production, Test, Dev environments. Also, remember that when they redisgned the pollers to be multi-threaded (one of the greatest things SW has done!!) you can now add WAY more weight to them than perviously in the past. My polling weights are 48% and 40% respectively with all NPM and SAM jobs running. I'm not sure if SAM monitors are figured into Element weight, but if not then it would be higher since I do some intense SAM polling from Web Apps to our entire Oracle/SQL clusters from production down to dev.

Not only that the servers I run SW on are a bit overkill to management, but in my mind there is never enough power!

bobby.alvarez

Sounds like a slick setup!! I'll have to work towards a similar configuration. Currently we have two Orion NPM / SAM instances, one on each US coast. When monitoring, we have two web dashboards to look at. I would love to have the single dashboard setup like you have and just have a poller feed in the one instance.

rjnicholson

It's a great set up and gets the job done very well.

One thing to mention since you have 2 instances of NPM and SAM already. Have you thought about just implementing the EOC console. This will give you the single management pane you are looking for. This might be a better solution for you since the money has been spent already on the current build you have. Unless you aren't in maintenance and about to refresh your contract and build then I would look at this solution as well as collapsing your current build into a single instance.

familyofcrowes

We are only CONUS so my MPLS stays here in the US. But, We are awtching over 16,000 elements, 700 applications, NTA at all sites, VNQM/IPSLA at all sites, UDT and NCM that hits all sites every day. We have 600 servers, 200 remote, 150 Riverbeds, and 2000 Cisco L2/# devies.

We use a primary Orion server (32 CPU, 32 Gb) and two VM based pollers (2CPU, 6Gb) and NCM runs on a separate server for now.

Our performance is excellent. There is more syslog traffic than SNMP traffic and we watch all L3 devices every 120 seconds. I have no issues with response time, even over our DMVPN sites that use cell cards and cable modems. Thresholds are mostly set at defaults.

We do not use EOC, but sounds intresting and could make it so that you have more freedom with two main systems.

But my experience is that a centralized system seems to work just fine, and if your anything like my IT department, the less to take care of the better considering all the different hats we wear. :-)

rharland2012

Thanks for the input, everyone! Much appreciated.