What are your performance stats?

Question

We do a lot of internal testing, but we're wondering how NPM is performing in the field.  What's your CPU and Memory usage like?  Please let us know.

What are you running on your server?

NPM?   How many elements?

NTA?  How many interfaces?

VoIP?  How many VoIP sites?

APM?  How many component monitors?

What does your box look like in terms of CPU and memory capacity?

What's your average CPU usage?  What's your average memory usage?

njoylif · Answer

All, after fixing a HUGE SQL issue (needed to enable AWE, which may or may not be a duh for most!) my environment is smokin'.  We have a separate SQL (clustered) environment with SAN (across 6 disks).
Once I enabled AWE with a minimum of 8GB RAM used by SQL, its doing very well.  Avg queue time is maybe 2.  SQL boxes are 4 dual core procs with 24GB RAM.  around 7.5GB RAM used, approx 5% cpu avg utilization on SQL.  This box is also running additional web server.

All nodes being status polled every 60 seconds, Statistics every 5 minutes as we have a real-time environment.
NPM Primary is running: 2 dual core, 4GB RAM, using ~20% CPU and 1.25GB RAM - Proceses NPM - 190MB, w3wp - 165MB, Netflow - 230MB, traps 30, scheduler 50, business layer - 50
       Polls per second (PPS recommended Status/Statistics) 148/57  99.55% polling success.
2nd Poller - same setup - using ~10-15% and 650MB, processes NPM - 180MB, job 30, business 50MB     PPS  139/55  99.63%
3rd Poller - same setup - using ~ 5-10%, 650MB, processes NPM - 150MB, job 30, trap 30, syslog 20, business 50MB    PPS 106/39  99.31%

Database Server192.168.x.xCatalogNetPerfMonDatabase EngineMicrosoft SQL ServerSQL VersionMicrosoft SQL Server 2005 - 9.00.3042.00 (Intel X86)
Feb 9 2007 22:47:07
Copyright (c) 1988-2005 Microsoft Corporation
Enterprise Edition on Windows NT 5.2 (Build 3790: Service Pack 2)Security ModelUser Name \ Password User NameSolarWindsNPM Connection Timeout20 seconds  Network Elements10109 ElementsNodes587 NodesInterfaces9511 InterfacesVolumes11 Volumes  Alerts1 AlertsEvents37446 EventsPollers19247 PollersPolling Engines3 Polling Engines

Retain Detail Stats7 DaysRetain Hourly Stats10 DaysRetain Daily Stats365 DaysRetain Events30 Day

savell · Answer

Denny,

We have approx 4300 nodes and 16000 elements spread over three pollers. Each poller is dual processor, dual core Xeon 3Ghz with 4Gig RAM. One processor core works much harder than the others - a sign of a single threaded design. Currently I would go for a faster CPU rather than multi core or multi processor machines for the pollers.

Our database is on a separate server, which also runs the additional web server for the bulk of users (30+ I think - but I can't measure this). Web performance is much improved for us when running on the database server.

The primary poller also runs VoIP (only 4 sites currently  - hub and spoke), APM (160 monitors) and NTA (only ad-hoc Netflow collection).

Memory is not an issue on the pollers - 5000 elements tend to utilise approx 450- 500meg for the main NetperfMonService and about 200meg for the SWJobEngineWorker process. The other SolarWinds related processes use 100-150meg combined.

The biggest memory usage problem we have is actually on a related web server process (dllhost) - this has a memory leak and keeps growing until we unload the process every 3-4 days (this has been an issue since version 8 - with a number records opened but no result thus far).

In terms of processor utilisation, the 'busy' core tends to average around 60-65% with the other cores trending around 10-15%.

We did some benchmarking ourselves when we were on version 8 and came to the conclusion that 5000 elements per poller was a safe number. I think we could push it some more (and I'm sure many others do), but that was a number we thought we could live with without undue stress.

When we did this benchmarking, I also felt that Network performance also played a part in limiting the number of elements we could support. While throughput on the FastEthernet adapter only averages 4-6Mbps with 5000 elements, I think the huge number of small network packets limits the throughout performance on the HP NIC's we use (and we don't seperate out the database traffic onto a different NIC). We didn't progress this theory - it would be interesting if anyone else has done testing in this area.

Dave.