(repost as the original got 'lost in translation'...) I'm looking for info from people running Orion with a lot of pollers and/or large databases (30G+.) Due to my own stupidity, I got to see what massive write blocks on the database looks like. This led me to look a little more closely at the forum to see what other people are doing. I have 11 pollers (don't shoot the messenger - I inherited...) and a separate DB currently at ~15G. The core question: go deep or wide with the pollers? Is it better to have more pollers that are lightly loaded or fewer pollers with greater load (and fewer processes writing to the DB.) * Thoughts on deep vs. wide? * How unwieldy are larger databases? Is there a 'cutoff' for when a DB becomes too big? * Is anyone else seeing avg disk queue lengths exceeding the 'max'. If so, what did you do to alleviate the issue? Some related questions: * Syslog: What kind of performance impact do you experience turning this on? Do you use only one poller for this or do you spread it out across multiple pollers? * SNMP Traps: What kind of performance impact do you experience turning this on? Do you use only one poller for this or do you spread it out across multiple pollers? * Custom MIB pollers: What kind of performance impact do you experience turning this on? Thanks in advance for any help or info. My DB and stats info is below. I'm looking to actually do 1 minute stat polls on part of the infrastructure but don't think the DB can handle it with the number of pollers and avg disk queue length just yet. Will Database Details Web ServerNT-CHNSADM09Operating SystemWindows 2003 Enterprise EditionOS Version5.2.3790Service Pack0.0 Database ServerNT-CHNSSQL01CatalogNetPerfMonDatabase EngineMicrosoft SQL ServerSQL VersionMicrosoft SQL Server 2000 - 8.00.2039 (Intel X86) May 3 2005 23:18:38 Copyright (c) 1988-2003 Microsoft Corporation Enterprise Edition on Windows NT 5.2 (Build 3790: )Security ModelUser Name \ Password User NameSolarWindsNPM Connection Timeout20 seconds Network Elements20452 ElementsNodes684 NodesInterfaces19725 InterfacesVolumes43 Volumes Alerts27 AlertsEvents81335 EventsPollers40552 PollersPolling Engines11 Polling Engines Oldest Event22-Apr-07 12:00 AMOldest Daily Traffic Stat22-May-06 12:00 AMOldest Hourly Traffic Stat22-Apr-07 12:00 AMOldest Detailed Traffic Stat15-May-07 12:00 AM Last Database ArchiveNeverNext Scheduled Database ArchiveNot ScheduledDatabase Archive TimeUnknown Retain Detail Stats7 DaysRetain Hourly Stats10 DaysRetain Daily Stats365 DaysRetain Events30 Days Historical Statistics Cisco Buffer Statistics145,861Detail recordsFrom15-May-07To22-May-07 06:29 AM 0To7 days ago41,635Hourly recordsFrom22-Apr-07To14-May-07 11:00 PM 8To30 days ago48,154Daily recordsFrom22-May-06To21-Apr-07 12:00 AM 31To365 days ago CPU Load Statistics3,840,535Detail recordsFrom15-May-07To22-May-07 06:29 AM 0To7 days ago257,009Hourly recordsFrom22-Apr-07To14-May-07 11:00 PM 8To30 days ago124,613Daily recordsFrom22-May-06To21-Apr-07 12:00 AM 31To365 days ago Response Time Statistics5,740,414Detail recordsFrom15-May-07To22-May-07 06:29 AM 0To7 days ago348,788Hourly recordsFrom22-Apr-07To14-May-07 11:00 PM 8To30 days ago130,624Daily recordsFrom22-May-06To21-Apr-07 12:00 AM 31To365 days ago Interface Traffic Statistics31,989,339Detail recordsFrom15-May-07To22-May-07 06:29 AM 0To7 days ago5,116,329Hourly recordsFrom22-Apr-07To14-May-07 11:00 PM 8To30 days ago2,374,179Daily recordsFrom22-May-06To21-Apr-07 12:00 AM 31To365 days ago Interface Error Statistics28,138,864Detail recordsFrom15-May-07To22-May-07 06:30 AM 0To7 days ago4,786,640Hourly recordsFrom22-Apr-07To14-May-07 11:00 PM 8To30 days ago2,256,655Daily recordsFrom22-May-06To21-Apr-07 12:00 AM 31To365 days ago

Hey NG, while you are waiting for your new toys study up on this link, it explains the ins and outs of SQL server performance tunning with very easy to understand instructions and down to earth reasoning. I just did the audit last weekend and man the Historical reports run just as fast as loading a Node Detail page! No more SQL query timeouts when I'm playing in Report Writer with raw SQL queries either.

What are you running for SQL? If your database is large (10GB or larger) you should read the following link Thanks for the link John - great resource for SQL 2005 info. I'm just in the process of ordering a separate SQL DB server & was only going to get 4GB of Memory - I think now I'll order 16GB instead, as I will be also be adding a second polling engine & doubling the number of elements being monitored.

Holy Cow!!! The manual maintenance you suggested has cut the DB size in half! Do you know what the difference is between the nightly maintenance and the 'manual' maintenance? The nightly maintenance does the statistical roll-up among other things, the manual maintenance only compacts the tables and rebuilds the indexs in the table selected, re-building the indexes is what "pounds" CPU(s) and disk... What are you running for SQL? If your database is large (10GB or larger) you should read the following link

Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Pollers: deep or wide?

willhepptemp

(repost as the original got 'lost in translation'...)

I'm looking for info from people running Orion with a lot of pollers and/or large databases (30G+.) Due to my own stupidity, I got to see what massive write blocks on the database looks like. This led me to look a little more closely at the forum to see what other people are doing. I have 11 pollers (don't shoot the messenger - I inherited...) and a separate DB currently at ~15G.

The core question: go deep or wide with the pollers? Is it better to have more pollers that are lightly loaded or fewer pollers with greater load (and fewer processes writing to the DB.)

Thoughts on deep vs. wide?
How unwieldy are larger databases? Is there a 'cutoff' for when a DB becomes too big?
Is anyone else seeing avg disk queue lengths exceeding the 'max'. If so, what did you do to alleviate the issue?

Some related questions:

Syslog: What kind of performance impact do you experience turning this on? Do you use only one poller for this or do you spread it out across multiple pollers?
SNMP Traps: What kind of performance impact do you experience turning this on? Do you use only one poller for this or do you spread it out across multiple pollers?
Custom MIB pollers: What kind of performance impact do you experience turning this on?

Thanks in advance for any help or info. My DB and stats info is below. I'm looking to actually do 1 minute stat polls on part of the infrastructure but don't think the DB can handle it with the number of pollers and avg disk queue length just yet.

Will

Database Details


Web Server	NT-CHNSADM09
Operating System	Windows 2003 Enterprise Edition
OS Version	5.2.3790
Service Pack	0.0

Database Server	NT-CHNSSQL01
Catalog	NetPerfMon
Database Engine	Microsoft SQL Server
SQL Version	Microsoft SQL Server 2000 - 8.00.2039 (Intel X86) May 3 2005 23:18:38 Copyright (c) 1988-2003 Microsoft Corporation Enterprise Edition on Windows NT 5.2 (Build 3790: )
Security Model	User Name \ Password
User Name	SolarWindsNPM
Connection Timeout	20 seconds

Network Elements	20452 Elements
Nodes	684 Nodes
Interfaces	19725 Interfaces
Volumes	43 Volumes

Alerts	27 Alerts
Events	81335 Events
Pollers	40552 Pollers
Polling Engines	11 Polling Engines

Oldest Event	22-Apr-07 12:00 AM
Oldest Daily Traffic Stat	22-May-06 12:00 AM
Oldest Hourly Traffic Stat	22-Apr-07 12:00 AM
Oldest Detailed Traffic Stat	15-May-07 12:00 AM

Last Database Archive	Never
Next Scheduled Database Archive	Not Scheduled
Database Archive Time	Unknown

Retain Detail Stats	7 Days
Retain Hourly Stats	10 Days
Retain Daily Stats	365 Days
Retain Events	30 Days

Historical Statistics

Cisco Buffer Statistics

145,861	Detail records	From	15-May-07	To	22-May-07 06:29 AM	0	To	7 days ago
41,635	Hourly records	From	22-Apr-07	To	14-May-07 11:00 PM	8	To	30 days ago
48,154	Daily records	From	22-May-06	To	21-Apr-07 12:00 AM	31	To	365 days ago

CPU Load Statistics

3,840,535	Detail records	From	15-May-07	To	22-May-07 06:29 AM	0	To	7 days ago
257,009	Hourly records	From	22-Apr-07	To	14-May-07 11:00 PM	8	To	30 days ago
124,613	Daily records	From	22-May-06	To	21-Apr-07 12:00 AM	31	To	365 days ago

Response Time Statistics

5,740,414	Detail records	From	15-May-07	To	22-May-07 06:29 AM	0	To	7 days ago
348,788	Hourly records	From	22-Apr-07	To	14-May-07 11:00 PM	8	To	30 days ago
130,624	Daily records	From	22-May-06	To	21-Apr-07 12:00 AM	31	To	365 days ago

Interface Traffic Statistics

31,989,339	Detail records	From	15-May-07	To	22-May-07 06:29 AM	0	To	7 days ago
5,116,329	Hourly records	From	22-Apr-07	To	14-May-07 11:00 PM	8	To	30 days ago
2,374,179	Daily records	From	22-May-06	To	21-Apr-07 12:00 AM	31	To	365 days ago

Interface Error Statistics

28,138,864	Detail records	From	15-May-07	To	22-May-07 06:30 AM	0	To	7 days ago
4,786,640	Hourly records	From	22-Apr-07	To	14-May-07 11:00 PM	8	To	30 days ago
2,256,655	Daily records	From	22-May-06	To	21-Apr-07 12:00 AM	31	To	365 days ago

Find more posts tagged with

sql

Database

Accepted answers

All comments

BryanBecker

We have a smaller implementation with about 6200 elements. We have 4 remote pollers and 1 NPM server. 2 of the 4 pollers are in the same data center as the NPM abd DB server. The other 2 pollers are in our backup datacenter a 1000 miles away. The only issue we see is the 2 pollers not local load a little slower and sometimes they will stop send data the the db and I have to restart the services to get them running again.

SW does not recommend this design but realitically they need to. Most larger companies have offices spread out. Our latency between the datacenters is less then 40MS.

jtimes

ELEVEN Polling Engines!!! What an admin nightmare, why so many? You can save allot of money and admin headaches by consolidating them down to five or six. The SLX polling engine can handle 6K elements each without getting strange; of course your servers need to be beefy. (max memory and processor)A deep Orion deployment is more efficient for cost and admin purposes. I couldn't imagine keeping up with nodes spread across 11 engines...Your disk queuing issues can be caused by the speed of the disk(s) in the DB server; I fixed mine by writing the DB to a SAN drive.

Your DB is in dire need of maintenance as well. There is no reason it should be that large. Back your DB up before you start. Your going to have to do manual maintenance via the DB Manager on the Orion server and it should be done during a maintenance window. Open the DB Manager and select your server, then your database, then right click and select database details. Go to the tables and sort them by size, then for every table over 1GB right click and select Compact/Repair. Note: This will take a while, my largest table is Interface Traffic - 6GB (takes about one hour for this to compact) I have been told by the developers that v8.1 will address large DB sizes and the maintenance issues. My DB floats between 16 - 25GB

You should consider archiving your DB, using the Orion nightly or weekly maintenance. I back mine up to the F: drive of the SQL server and then that file is also backed up by an external Netbackup job.

I have four polling engines and one HSB server:

Web Server
Operating System Windows 2003 Enterprise Edition
OS Version 5.2.3790
Service Pack 1.0

Database Server
Catalog NetPerfMon
Database Engine Microsoft SQL Server
SQL Version Microsoft SQL Server 2000 - 8.00.2162 (Intel X86)
Sep 30 2005 19:59:14
Copyright (c) 1988-2003 Microsoft Corporation
Enterprise Edition on Windows NT 5.2 (Build 3790: )

Security Model
User Name
Connection Timeout 30 seconds

Network Elements 19406 Elements
Nodes 6717 Nodes
Interfaces 12512 Interfaces
Volumes 177 Volumes

Alerts 36 Alerts
Events 40372 Events
Pollers 37202 Pollers
Polling Engines 5 Polling Engines

Oldest Event 22-Apr-07 12:00 AM
Oldest Daily Traffic Stat 23-Nov-06 12:00 AM
Oldest Hourly Traffic Stat 22-Apr-07 12:00 AM
Oldest Detailed Traffic Stat 07-May-07 12:00 AM

Last Database Archive 22-May-07 01:00 AM
Next Scheduled Database Archive 23-May-07 01:00 AM
Database Archive Time 01:00 AM

Retain Detail Stats 15 Days
Retain Hourly Stats 30 Days
Retain Daily Stats 180 Days
Retain Events 30 Days

Syslog: What kind of performance impact do you experience turning this on? Do you use only one poller for this or do you spread it out across multiple pollers?

No impact, but we don't send anything below Notice level syslog messages. I have also set a rule in the Syslog server to ignore/trash syslog messages that are workstations joining or leaving the network. Every Network device sends syslog messages to one Orion Server and to the Network Automation Server.

SNMP Traps: What kind of performance impact do you experience turning this on? Do you use only one poller for this or do you spread it out across multiple pollers?

No impact here either. Once again I only send them to one poller (the Orion Master server.) Not that many traps defined, mostly for the old equipment...

Custom MIB pollers: What kind of performance impact do you experience turning this on?

Custom MIBS... errrr, I have stopped developing anything with them until the next release of Orion; but what do have on, environmental monitors etc, works. The impact felt here is time! It takes a while to get to the MIB your looking for, and just like surfing the World Wide Wait (web), I get distracted in the MIB tree then realize I have forgotten what I was looking for in the first place... But that is just me

willhepptemp

> ELEVEN Polling Engines!!! What an admin nightmare, why so many?

Heh heh, like I said - I inherited. I'm glad to hear your target number of 5 or 6 as that's what I was thinking in my head but wanted some validation before I went nutty. (I knew I was in trouble when I called in for support a while back and the first thing the tech said on hearing I had 11 polling engines was "WTF!?" I'm sure the sales guy was really happy with us...)

> The SLX polling engine can handle 6K elements each without getting strange; of course your servers need to be beefy. (max memory and processor)

The pollers have a minimum of 3.6 Ghz processors and 4 Gig of RAM each. They're not really breaking a sweat just now!!

> A deep Orion deployment is more efficient for cost and admin purposes.

No doubt. I was wondering if it would also be a bit more gentle on the DB with few pollers streaming the data (less contention.) Don't know if that logic makes real world sense or not.

> Your disk queuing issues can be caused by the speed of the disk(s) in the DB server; I fixed mine by writing the DB to a SAN drive.

My admin is looking further into the disk issues to see what can be done to improve performance. I trust his judgement but I wanted to get a feel for what other people are doing (i.e., not all apps behave 'badly' in the same way...)

> Your DB is in dire need of maintenance as well.

I run the maintenance nightly, actually. I also do a backup and keep two days worth from the SQL Manager application. Maintenance job runs a little more than 3 hours and the compact can run in about 2. The states above are taken from the DB less than 24 hours since the last maintenance and compact.

> You should consider archiving your DB, using the Orion nightly or weekly maintenance.

Do you mean that as a separate function from backing up? Since I'm doing the roll-ups/maintenance and backups nightly, I figured I was covered...

Thanks for the response and all the info - one last question: What are your polling intervals for statistics?

Thanks again,

Will

jtimes

Yes use the database backup function in Orion to save the DB to another drive (and then back that up) IMO: This works better than the SQL maint process for DB backups for Orion.

Polling intervals for statistics collection:

Nodes = 10 minutes

Interfaces = 5 minutes

Volumes = 15 minutes

Just remeber that on charts and reports, you can only get a 5 minute sample period, so polling any smaller than 5 minutes is a bit of overkill.

willhepptemp

> Just remeber that on charts and reports, you can only get a 5 minute sample period, so polling any smaller than 5 minutes is a bit of overkill.

While that's true, the interface will also give you candle graphs when there's more data available than it 'natively' displays. If I could get away with polling every second and getting real data, there are certain parts of the network where that info would be valuable (quantitative trading.)

Thanks again,

Will

Network_Guru

Well if you're only using 5 or 6 pollers now, do you mind sending a couple of the extra's my way ;-)
<hehehe>Perhaps you can sell the rest on e-bay</hehehe>

I'm running close to 8000 elements on my single server with integrated SQL DB & I'm pushing the performance limits. DB size is between 8 & 10GB & is kept in check using the scheduled nightly maintenance.

As for the statistics polling interval, that is what's really killing your DB.
If you must, then only tweak around 5% of the polls to collect at 1 minute intervals. Leave the rest at 5 minutes.
Be careful with this, as you can kill your devices with excess SNMP queries. Orion is really not designed for real-time stats collection - use the Toolset bandwidth gauge for that.
If you are polling a 6500 with a couple of hundred interfaces, it may take more that a minute to collect all the stats (esp. if the box is busy).
In the meantime, you will be starting another poll before the last one has finished.
BTW - you can get the graphs to display in 1 minute intervals by tweaking the URL used to display the graph.
Just change the 5m to 1m.

I've actually added some custom buttons on all my graphs to automatically display the stats for 7, 30, 90 & 365 days with improved granularity than the 2 out of the box buttons, included in the default install. There's no reason you can't add a button for the past hour with 1 minute granularity.

I'll post up a pic & the code, if anyone is interested....