This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Netflow v3 Service Crash

My Netflow v3 service crashed today.


 Event Type: Error
Event Source: NetFlowService
Event Category: None
Event ID: 0
Date:  4/30/2008
Time:  9:53:58 AM
User:  N/A
Computer: ORION1-CORP
Description:
Critical error in NetFlow listener: Exception of type 'System.OutOfMemoryException' was thrown.


For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


Event Type: Error
Event Source: NetFlowService
Event Category: None
Event ID: 0
Date:  4/30/2008
Time:  9:54:01 AM
User:  N/A
Computer: ORION1-CORP
Description:
Exception: error occurred during packet processing. Exception of type 'System.OutOfMemoryException' was thrown.


For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


Event Type: Information
Event Source: NetFlowService
Event Category: None
Event ID: 0
Date:  4/30/2008
Time:  9:54:09 AM
User:  N/A
Computer: ORION1-CORP
Description:
NetFlow Receiver Service [ORION1-CORP] Stopped


For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


  •  Are you actually seeing that the system is running out of memory?  What about the service growing in memory size during the day?  Just need to determine if this might be a memory leak issue or just an exception from some resource that is running out of resources.  Are there a lot of flows coming to the receiver?  If so, about how many?

  • This may be an issue that we saw in the beta where the SQL Server wasn't able to keep up with all the NetFlow traffic that we were trying to store.  The database bottleneck was causing NTA to queue up more and more data that it was trying to feed to the database, but over time it caused the NTA service to grow and grow in memory size until the system finally ran out of memory.

  • Memory is good, 2 gig available.

    I see the service build up to 530meg, then tear down to 380meg. But it keeps getting higher each build up.

    NetFlow Sources

    620 interfaces

    Process - % Processor time - NetflowService - Avg. 71

  • David,


    This could be the problem, netflowservice.exe is now 836meg. 


  • David,


    I can't use Neflow and I'm still waiting for support to call me.


    What is going on? 


     


  •  We just received your diagnostics and from the first indications it appears that the transaction log is full and subsequently kicking off the remaining errors. The DEV team should have a better understanding of what is causing this shortly.

  • Yes, reviewing your diagnostics, I see that in your Summary2 table, you have over a billion rows.  Because of the issue in 2.2.1 where table in Summary2 never got collasped into Summary3 and hence never was groomed out of the system, this table has grown to this size.  I've given your tech support rep. instructions and T-SQL on how to clean this up.  Keep in mind that the T-SQL that I sent will groom all data from all 3 summary tables that is older than 7 days.  Running that will be a big hit on your database.  I've written the script so that you can adjust those numbers.  So that you don't impact your transaction log greatly, you may first may to delete all data that is 180 days or older.  Then adjust the numbers down and run the script again.  Eventually adjust that down enough to your desired 7 days or older.


     


    Thanks,


    David


  • David,


    The grooming did not solve the problem. NetflowService continues to build up memory and crash.


  • Can you open Microsoft's PerfMon by going to the Start\Run and type in PerfMon.  Then add a new counter called SolarWinds\Raw Packet Queue.  Start the service and see it if keeps rising as the memory goes up.  This will indicate if the NTA service is receiving so much flow traffic that it can't send it fast enough to the SQL database.


     


    Thanks,


    David