nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Missing Data from Orion polling

pwatkins

Hi,

I have been experiencing issues where there are siginificant gaps in statistics gathering. Primarily on interfaces, but, I have seen it on CPU and memory gathering also. Alerts seem to work pretty good as far as I know.

I worked with Solarwinds Tech Support on this last year to no avail. I had separate polling and SQL servers, moving everything onto one server seemed to improve things somewhat with only occasional data loss.

We just installed brand new HP DL380G5 servers and installed Orion v8.0.1 on separate servers again - 1 polling and 1 SQL server. We are running on Windows 2003 R2 and SQL Server 2000 SP4. Each server has 4 GB of memory, a total of 8 CPU's in Task Manager on each server (3.2 Ghz Dual Core) and SAS Drives. These servers are barely breaking a sweat right now.

We are polling 6646 elements and it seemed to work O.K. until I added the Wireless Application pack. Things have gotten much worse again. It takes forever to Orion System Manager to load and there is a direct correlation to that and when there is data loss. We monitor about 520 AP's, 220 routers, 500+ switches and about 100 servers.

I am using the default polling settings, I have ran the Polls Per Second tuner. It recommends 132 per second for Node and Interface Polls and 50 per second for Max Statistics Collections.

I am at a loss, this has been a year long battle. Are we just trying to do too much with a single polling server? Do I need to split the load to another polling engine?

Any suggestions or if anyone needs more details, please let me know. I'd love to get this resolved, thanks!

Paul W.

Find more posts tagged with

Accepted answers

All comments

Network_Guru

I feel your frustration Paul...

<rant on>
Lately I've seen gaps in certain graphs as well (CPU, Interface util.etc.).
I'm still running Version 7.x right now, but what I have found is that the weekly DB maintenance built into Orion has a shrink DB script that runs every Sunday morning after the re-indexing takes place.
When I come in Monday morning, Orion is dog slow because the DB has 0mb free space. It is set to grow automatically (by %10), but it seems to either not grow at all, or takes too long to add more room.
I end up manually expanding the DB every week, which seems to fix the gaps & performance issues.
I see errors in the Orion server's Eventlogs under Solarwinds.net events.
They are mostly from the Syslog service unable to open or write to the DB.
I've sent these error messages to support@solarwinds.net about a month ago, but have yet to receive a reply.

Check your maintenance log files in the Orion directory (you should have 14 days worth) to see if your DB is being shrunk on Sunday morning (mine starts after the regular maint. job is done ~ 3:10AM & doesn't finish until after 8AM).

I've tried editing the .maintenance files to stop the shrink, but it seems these scripts only apply to the default built-in SQL DB & not the Enterprise version I am running. For the life of me, I can't find the script or stored procedure that runs the nightly & weekly maintenance anywhere. Solarwinds has done a great job hiding these, even my DBA can't find them, & is at a total loss on how to fix this.

The strange thing is, I've only had problems with this for the last couple of months. I suspect something was changed when they added the Syslog Engine & tables to the DB...
</rant off>

There I feel much better now... :-/

-=Cheers=-
NG

~seek & ye shall find---> [url="http://www4.solarwinds.net/SolarWindsForum/search.asp"]

<---no need to wait for answers~

pwatkins

Hey NG,

This is interesting. We started experiencing the data loss under 7.8x of Orion and it was almost exactly when we installed the Orion to support syslog.

Of course when I upgraded to Orion v8.x, I pulled the same database across with the same tables.

How do I manually increase the size of the database as you mentioned in your posting?

Thanks,

Paul W.

tpula

We have been battling the same issue since 7.8. I have worked with SW senior techs at length with no resolution. I have rebuilt our Orion environment from scratch but the problem persists. It doesn't matter what polling engine settings we use. We can go hours without data. I have dedicated app and db servers with plenty of horsepower. I question the Wireless Application pack as well.

Todd

FormerMember

We are having what seems to be a similar issue with v7.8.5. We have 2 polling engines and every day or so one of the pollers (mainly the secondary) fails in one of two ways. Either the netperfmonservice just stops or it "hangs" and stops updating the database on the SQL server. When the service stops, in which case Windows 2003 Server posts a Windows System event, it sometimes is restarted but, in the majority of cases, it isn't restarted and we have to kill and restart the service. When the service "hangs" the poller is still updating ICMP stats but SNMP data such as CPU, memory, interface and volume stats are no longer recorded (Last Database Sync). We have captured each incident with our sniffer and see a significant change and dropoff in traffic between Orion and SQL when the "hang" condition occurs. I am speculating at this point that the limited volume of SQL traffic still passing is the ICMP stat data. The sniffer files were forwarded to SW last week but we haven't received any feedback as yet.

We have enabled poller debugging but the info doesn't reveal anything consistent like failing on the same function, node or interface, etc. The last entries each reveal different actions with different nodes.

When examining the Windows Application event log and the Orion NetPerfMonService.log you will see a huge volume of Timer Tick errors generated and they start about 1-2 hours after the "hang" occurring almost every second until the service is killed and restarted. SW tells us that they are .NET Framework errors but don't provide any significance. DAH? SW...it's a clue that the application is SICK and NOT WORKING RIGHT!!!

This problem has been ongoing for 2 1/2 months now and SW hasn't been much help or responsive

tpula

We ran in to the same service hang issue after SW tech support enabled the polling debugger. Once the debugger was disabled, the problem went away.

Todd ***
Network Engineer - CCNA/CCDA/CCNP
The Stanley Works

Network_Guru

You can manually increase the DB size using SQL Enterprise Manager. In the DB Properties > Data Files, change the Space allocated(MB) of your NetPerfMOn DB file.

I upped mine to 12GB last week.
I checked it this AM & it was back down to 9.1GB.
I've lost over a days worth of data since the DB maintenance on Sunday morning.
Once again & upped it to 12GB this morning & now it shows 1.9GB free & polling has returned to normal.

There is a hidden script somewhere which is running a job every Sunday AM which shrinks(compacts) the DB, leaving now free space. Even though the DB is set to grow automatically, it seems that it does not grow when there is no more free space & the polling just stops.

Here is the last few lines of my NetPerfMOn-Maint-20070218.log file showing the DB being shrunk:


2/18/2007 3:09:21 AM        Summarizing into Hourly Volume Usage Summarizing VolumeID 812
2/18/2007 3:09:21 AM        Summarizing into Hourly Volume Usage Summarization completed sucessfully for VolumeID 812
2/18/2007 3:09:21 AM        Summarizing into Hourly Volume Usage Archive completed sucessfully for VolumeID 812
2/18/2007 3:09:21 AM        Step Commited Sucessfully
2/18/2007 3:09:21 AM        Deleting Old Events
2/18/2007 3:09:21 AM         
2/18/2007 3:09:23 AM        Step Completed Sucessfully
2/18/2007 3:09:23 AM        Adding Event for Deleting Events
2/18/2007 3:09:23 AM         
2/18/2007 3:09:23 AM        Step Completed Sucessfully
2/18/2007 3:09:23 AM        Re-indexing Interface Traffic Table
2/18/2007 3:09:23 AM         
2/18/2007 3:46:22 AM        Step Completed Sucessfully
2/18/2007 3:46:22 AM        Re-indexing Interface Errors Table
2/18/2007 3:46:22 AM         
2/18/2007 4:03:04 AM        Step Completed Sucessfully
2/18/2007 4:03:04 AM        Re-indexing Response Time Table
2/18/2007 4:03:04 AM         
2/18/2007 4:11:57 AM        Step Completed Sucessfully
2/18/2007 4:11:57 AM        Re-indexing CPU Load Table
2/18/2007 4:11:57 AM         
2/18/2007 4:18:00 AM        Step Completed Sucessfully
2/18/2007 4:18:00 AM        Re-indexing Cisco Buffers Table
2/18/2007 4:18:00 AM         
2/18/2007 4:18:11 AM        Step Completed Sucessfully
2/18/2007 4:18:11 AM        Re-indexing Volume Usage Table
2/18/2007 4:18:11 AM         
2/18/2007 4:18:22 AM        Step Completed Sucessfully
2/18/2007 4:18:22 AM        Shrinking the NetPerfMon Database
2/18/2007 4:18:22 AM         
2/18/2007 5:33:55 AM        Step Completed Sucessfully
2/18/2007 5:33:55 AM        Checking SQL Database Size
2/18/2007 5:33:55 AM         
2/18/2007 5:33:55 AM        Step Completed Sucessfully
2/18/2007 5:33:55 AM        Adding Event for Completing Maintenance
2/18/2007 5:33:55 AM         
2/18/2007 5:33:55 AM        Step Completed Sucessfully
2/18/2007 5:33:55 AM        Archive Complete
2/18/2007 5:33:55 AM        ----------------

I've also edited the NetPerfMon-Compact.Maintenance file as per this post;
www4.solarwinds.net/.../topic.asp
in an attempt to stop the DB from being compacted, but I believe this only pertains to the built-in MSDE DB file.


<!-- Remarked out the following line due to issue with the DB growth after the weekly compaction (NG_6-Feb-07) -->
	<!-- Post SQL="DBCC SHRINKDATABASE('${Database Name}',0)" ContinueOnError="True" Comment="Shrinking the NetPerfMon Database"/ -->
@VERSION like '%Desktop%' 
IF FILEPROPERTY(FILE_NAME(FILEGROUP_ID ('Primary' ) ),'SpaceUsed') / 2621 > 80
INSERT INTO Events ( EventTime, Acknowledged, Message, EventType, NetworkNode, NetObjectType, NetObjectID ) VALUES ( GETDATE ( ), 0, 'Database Size is reaching 2GB limit for SQL Desktop.   The maximum size for SQL Desktop is 2 Gigabytes.  Current file size is ' + CAST(FILEPROPERTY(FILE_NAME(FILEGROUP_ID ('Primary' ) ),'SpaceUsed') * 8096 AS varchar(15)) + ' bytes.  You may want to lower the polling rate, decrease the retention period, or upgrade your SQL Server.' , 1002 , 0, '',0 )

" ContinueOnError="True" Comment="Checking SQL Database Size"/>
</Maintenance>

In any case, I am renaming this file, so that it cannot run next Sunday - I'll see what happens....

We should all send our issues & log files to SW development as per this post;
www4.solarwinds.net/.../topic.asp

-=Cheers=-
NG

~seek & ye shall find---> [url="http://www4.solarwinds.net/SolarWindsForum/search.asp"]

<---no need to wait for answers~

pwatkins

I contacted Solarwinds and obtained a trial license for an additional polling engine and since I have added a second poller and distributed the load, my problems have disappeared. I have about 3800 elements on my big polling server and about 2900 elements on my small polling server. We have the Wireless and Application packs licensed and I just enabled the Wireless Application Pack. We'll see how it goes with that.

I think Solarwinds will need to adjust their system requirements as they start adding on all these additional modules and services, because I think 6700+ elements on one server was just too much, no matter how big it is.

mcobb

quote:Originally posted by pwatkins

I contacted Solarwinds and obtained a trial license for an additional polling engine and since I have added a second poller and distributed the load, my problems have disappeared. I have about 3800 elements on my big polling server and about 2900 elements on my small polling server. We have the Wireless and Application packs licensed and I just enabled the Wireless Application Pack. We'll see how it goes with that.

I think Solarwinds will need to adjust their system requirements as they start adding on all these additional modules and services, because I think 6700+ elements on one server was just too much, no matter how big it is.

They recommended the same thing to me but the new poller did not help, my services still stop randomly because of some .net errors and what they considered overloading of the polling servers.

btp

AOL to that; we bought a brand new Dell blade-rack with quad-core processors, bought another polling-engine, and the same thing happens. Fresh after reboot everything it's smooth as linen. I upgraded to latest version on April 19th, and today I have responsetime graphs with one dot every hour instead of every 5 minutes. And I have done the polling-engine tuning.

So sad... Maybe one should downgrade?

--
Bjørn Tore