Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Ideal Solarwinds Deployment

I would like some advice on the idea deployment for my company. We currently have 7 datacenters located throughout the US, but primarily in the NE. Our main DC monitors itself, and several of the smaller centers, while 2 of the larger ones have their own independent servers that monitor everything within their networks. We currently use EOC for some of our colo customers so they can log in and see a universal interface.

Our main DC currently monitors 1200 nodes, with 2100 volumes and 3000 interfaces, while the 2 separate instances are a little lower than those totals. Would we be benefited optimally or financially to move to a single overall instance of Orion and just deploy remote pollers in the 2 larger DC's?

I know this is a vague question, but i'm try to get opinions on the matter before I start investing huge amounts of time and energy when others might have useful insight. Thank you!

Find more posts tagged with

Accepted answers

All comments

CourtesyIT

rspotswood‌, This is kind of tricky I know. The Admin Guide does give information about what the server requirements should be, but unfortunately there is no "Magic" Build for a single Orion Server. I have worked in several environments over the years and given what you have provided us it sounds like that standard build would work just fine for you. If you have the ability to "grow" your resources if needed then I would start with the base build specs and go from there.

Since this is a key topic for those out there, if you could keep us informed on your progress we would appreciate it. Please include Orion Server, SQL Server, and Polling Engine Server spec, along with node counts on all and database sizing and architecture.

All the Best.

jkuvlesk

Hi rspotswood. This is a new video and should provide you additional guidance. http://players.brightcove.net/702974249001/414def77-c0d8-4e6e-b5cc-7be509c42519_default/index.html?videoId=4671589970001

jeff.stewart

How much bandwidth and what is the latency between the remote locations? While the Additional Polling Engines support remote-ability there are some factors to consider. Also, which modules would be running in these remote locations?

agusst

It really depends on what components you're going to use...

My current environment isn't nearly as large as yours...we have about 600 nodes, 2400 volumes, 1000 app monitors. The majority of the systems are in 2 main DC's with a handful of systems at remote sites. We only use Orion and SAM in this environment...we have one Poller and one SQL server.

My last gig we had about 7000-ish nodes....I can't remember the volumes or app monitors...50% of the systems were in one data centers...the remaining 50% were split between about 6-7 other DC's and several remote sites. We used SAM, NPM, IPAM and a few other components......In this environment we had 3-4 Pollers and two SQL servers in our main data center.

Not sure if this is helpful or not...but it gives you an idea of what I'm doing / have done.

ecklerwr1

You can't really put an additional poller at remote location without anything else... it has to talk to the database and talking to the database over the WAN isn't really practical. You really need to either stack pollers in one main site and monitor everything remotely or use multiple instances.

rschroeder

I think jeff.stewart‌ may have hit the most important question: What's the available bandwidth between the locations, and what's the latency. If you have fat enough pipes, I'd consider building a main NPM instance and using local pollers at each DC that report to the main NPM.

I had three separate instances of NPM, each doing two data centers, each pair of DC's about 150 miles from the next pair--the result of several mergers.

Local polling, local dedicated databases, all monitored by EOC. That was NOT a good way to go.

EOC degraded our team's ability to respond properly and in a timely manner due to its slow ability to display changes from the remote pollers, and also due to regional boundaries and "turf" responsibilities. Instead of being one team managing all regions, we ended up split in two--the bigger East region doing East's work, and West doing their own work plus that of the Central region.

Our WAN between these sites is a resilient MPLS solution with dual 1G pops at each region. Central and West are about 7 and 11 ms away from East.

We saved money by dropping EOC and converting the NPM instances in West and East into pollers that report to a new main NPM instance in East. Logging, NTA, NPM, and NCM all are done at the local regional pollers, which updates East's NPM.

We were able to remove the EOC license for a nice savings, and that simplified it to look like one organization instead of three. Our service response times and customer satisfaction improved.

ecklerwr1

You are reading and writing to the SQL server over the WAN? Most MPLS WAN's i've seen any further than a little way a way that wouldn't be possible... there's no way to cheat distance no matter how fat your pipes are it's physics. If you's are just a few miles down the road maybe but I have sites across the US and in other places around the world. There's zero way a poller could write the SQL server across the wan at most of these sites. If you have SQL Server at each site... that's different. I've yet to see anyone get away with something like that but maybe you're a first?

What I have seen people do it this...

convert from multiple instances of NPM reporting up to EOC to one massive instance with stacked pollers. From my experience much of your performance comes from reads/writes fast to the database more than anything else.

Perhaps you were blessed with close to theoretical latency... most aren't in the real world although things are getting better.

RichardLetts

I've been tasked with relocating one half of our FOE install from a location that is rtt min/avg/max/mdev = 0.380/0.496/0.671/0.060 ms

to a location that is rtt min/avg/max/mdev = 8.392/8.488/8.670/0.125 ms

i.e. about 8 milliseconds further away -- this site would only be active during disasters (when I would also be running the database there) or application maintenance on the main cluster (when the database would remain here)

[i.e. not in earthquake range -- the round-trip distance is ~560 miles, which accounts for ~5ms of the delay and the additional network hops for the other 3ms]

Do you think that is too much delay?

CourtesyIT

Anyone have some diagrams to depict what they have done. This is a really good conversation and would like to get some artwork on it.

CourtesyIT

rschroeder‌ and ecklerwr1‌, I am trying to calculate the bandwidth saving of having stacked pollers opposed to deployed pollers. Can you describe/define a little better. I am sitting on the fence about a solution for a global architecture and trying to come up with some solid pros and cons for each.

ecklerwr1

Redundancy for some things could be one although I'm not sure it matters that much anymore. With virtualization of everything (including SQL Server) I don't have much reliability issues any more. I have been monitoring wan sites from multiple locations ie. overlap between pollers. I also have EOC license still although the old one I rarely use (I'm hoping new version based on NPM will be better) but... I like you am considering all my remotes being brought back here to my main site. I currently have two stacked pollers now at my main site. (I'd increase this if I scrap the remotes)

Let me get this straight is anyone here actually running an additional poller without NPM at a remote site miles away? I hear Richard thinking of moving from sub ms to around 8ms latency to the SQL server... For as long as I can remember we didn't even want to have anything but a dedicated physical SQL Server right next to NPM until very recently. I've been virtualizing it since before it was supposed be ok and it's worked fine but... nothing in NPM or it's modules functions well at all without primo SQL server connectivity ie. it will never run any better than it's access to the database period. Also initial attempts to virtualize SQL for some was a disaster. This has always been the weak point. I think moving NTA to it's own noSQL database was another step in improving this.

I've always had sub ms access to SQL... I've never considered pushing this because I always assumed it would greatly hurt performance but... now I want to know if anyone has successfully even tried this? Everything the main NPM server does is all about the database so...

jeff.stewart

"Let me get this straight is anyone here actually running an additional poller without NPM at a remote site miles away?" Yes, we have customer doing this today and added some functionality to the APEs to help support this model. Additionally, if you look at the WWWO for NPM you'll see a "Remote APE" listed in the ongoing section. This is should continue to help the support of APEs that are not in the same location as the DB.

stevenstadel

We are reading and writing SQL over a WAN across country. We operate two datacenters 3,500 km apart with a single SQL instance. The link between the two DCs operate on average 40-45 ms latency on a 660 Mbps MPLS link. All our inter-datacenter traffic shares also shares this link.

Getting rid of EOC to a real single pane of glass was the best thing we would have done administratively.

We have 2 additional polling engines in our eastern DC monitoring 750 Nodes / ~8000 elements. The eastern DC connects back to our primary installation in the western DC running 5 additional polling engines 3000 Nodes / ~30000 elements.

We are running SAM, IPAM, SRM, NTA, NCM, NPM.

Take a look at

Centralized Deployment with Remote Polling Engines in

http://www.solarwinds.com/documentation/Orion/docs/ScalabilityEngineGuidelines.pdf

Latency should be below 300 ms.

rschroeder

There's a local SQL server in each region--no WAN traffic except for reporting stats to the main NPM instance. My apologies for not making this clear previously, and thanks for calling it out.

rschroeder

As someone previously caught, I have local SQL servers in each region. Our regional pollers simply report stats & make beautiful graphs with the main NPM solution, along with sending fast alerts.

rschroeder

CourtesyIT‌

I was mistaken about the SQL servers--all my remote pollers send their SQL over the WAN to my East NPM deployment--and it works quite well.

Orion pollers aren't killing our WAN with bandwidth or alerts or Netflow--not in any way.

There is no problem with the amount of WAN utilization by Orion--the three regions are connected by dual-path resilient 1G fiber, for an effective 2G in each direction. Hitless failovers, too, I'm happy to say, so the WAN provider or my team can take down one of the two routers at each site for replacement or upgrade maintenance and all we've done is temporarily decreased our resilience and throughput.

rschroeder

Disregard my comment here, ecklerwr1‌; I was mistaken. All my regional pollers report SQL and everything else back to my main East data centers.

I don't know why it works well here, or why others have had some issues with distance. My West and Central sites run about 7 ms and 11 ms in round trip latency from/to my East NPM instance.

It's very satisfactory--even for my team members that site 300 miles away.

It wasn't satisfactory when we used EOC.

CourtesyIT

Ok folks, here is one for you.

I like rschroeder‌ idea and case study. I have engineers all over the planet. 5 people / 50 nodes in Asia, 5 people /100 nodes in Hawaii, 15 people / 450 nodes in East Coast, and 5 people / 125 nodes in Europe.

I wanted to ask about additional webservers for each location.

rschroeder

I think a world-wide design should be leveraging Solarwinds' expertise to ensure the right solution is applied. What works well for my environment, which spans only three states, may not be the right deployment for a successful solution when covering multiple continents.