11 Replies Latest reply on Sep 2, 2015 11:27 PM by muheet.hajela@contractor.xlgroup.com

    Solarwinds Data Processor

    lekeosi11

      I'm currently trying to fix an Orion Additional Polling Engine where the Solarwinds Data Processor service stops with the following error message:

       

      Application: SolarWinds.DataProcessor.exe

      Framework Version: v4.0.30319

      Description: The process was terminated due to an unhandled exception.

      Exception Info: System.OutOfMemoryException

      Stack:

         at System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)

         at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)

       

       

      We've got 3 additional polling Engines and a main poller. The problem affects only one polling engine. The polling engines have the same hardware specifications.

       

      The polling rates on the Polling engines are around 80%. The polling engines have been working at this rate for the past few months but the problem with this poller only started in the last two weeks (no associated config changes).

       

      I've run through the process described here SolarWinds Knowledge Base :: Collector Data Processor and Collector Polling Controller start and stop intermittently but this did not resolve the problem.

       

      Does anyone have any ideas what else to check?

       

      Thanks

       

      L

        • Re: Solarwinds Data Processor
          sean.martinez

          The error is saying that the system did not have enough available RAM left to continue running the service. I would recommend to increase the available RAM by 2GB to handle the load of the Services. The Data Processor service is responsible to sending the polling and statistics information from MSMQ over to the SQL Server.

          1 of 1 people found this helpful
          • Re: Solarwinds Data Processor
            RichardLetts

            open a support case. Because the Solarwinds applications are only 32-bits the most they can address is 2GB; the last time this happened to me they had to provide a fixed DLL.

            1 of 1 people found this helpful
            • Re: Solarwinds Data Processor
              stefanIT

              Hi,

               

              Did this get resolved? If so, would you mind sharing what the resolution was?


              Thanks,

              Stefan

              • Re: Solarwinds Data Processor
                Vinay BY

                I agree with you on The polling rates on the Polling engines are around 80% , but then can you move few nodes onto a different poller and check if it helped you.

                 

                The polling engines have been working at this rate for the past few months but the problem with this poller only started in the last two weeks

                Were nodes be added onto this poller (last few weeks) ? If so as mentioned above please move few nodes onto a different poller if possible and check for the same.

                Another thing i would like to add on would be , did you by any chance change polling method on the nodes existing on this poller (I mean SNMP <-> WMI)?

                  • Re: Solarwinds Data Processor
                    irae@saccourt.ca.gov

                    My polling rate is only at 25% and I also get this Error.

                    I opened up a support case but they seem to this its a DB issue because I also see

                     

                    System.Data.SqlServerCe.SqlCeException (0x80004005): Not enough storage is available to complete this operation.

                     

                    In the dataProcessor.log.  But the Out of Memory Error shows up before the SQL Not enough storage space.

                     

                     

                     

                    2015-01-29 16:31:25,081 [14] ERROR SolarWinds.Collector.Workflow.FlowStorageProvider - System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

                       at System.IO.BinaryReader.ReadBytes(Int32 count)

                       at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadArrayAsBytes(ParseRecord pr)

                       at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadArray(BinaryHeaderEnum binaryHeaderEnum)

                       at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()

                       at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)

                       at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)

                       at System.Messaging.BinaryMessageFormatter.Read(Message message)

                       at SolarWinds.Collector.Queue.CompressedMessageFormatter.Read(Message message)

                       at System.Messaging.Message.get_Body()

                       at SolarWinds.Collector.Queue.MsmqQueueController.EndReceive(IAsyncResult asyncResult)

                       at SolarWinds.Collector.Workflow.FlowStorageProvider.OnItemArrived(IAsyncResult result)

                    2015-01-29 16:40:58,077 [STP SmartThreadPool Thread #86] ERROR DataPreProcessing - Processing result: PollerResultWrapper: PollerType=N.Memory.SNMP.CpqHostPhysicalMemory, NetObject=N:367 ErrorCode=FFFFFFFF, ErrorMessage=Unable to compute memory for 172.19.58.36, cpqHoPhysicalMemorySize.Count:0 != cpqHoPhysicalMemoryFree.Count:0 or no values returned

                    2015-01-29 16:41:24,815 [27] ERROR SolarWinds.Collector.Workflow.FlowStorageProvider - System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

                       at System.IO.MemoryStream.set_Capacity(Int32 value)

                       at System.IO.MemoryStream.EnsureCapacity(Int32 value)

                       at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)

                       at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)

                       at System.IO.Stream.CopyTo(Stream destination)

                       at SolarWinds.Collector.Queue.CompressedMessageFormatter.Read(Message message)

                       at System.Messaging.Message.get_Body()

                       at SolarWinds.Collector.Queue.MsmqQueueController.EndReceive(IAsyncResult asyncResult)

                       at SolarWinds.Collector.Workflow.FlowStorageProvider.OnItemArrived(IAsyncResult result)

                    ...

                    ...

                    2015-01-29 19:35:02,473 [35] ERROR SolarWinds.Collector.Scheduler.ScheduledTask - ActualizeInvetoryManager threw an exception.

                    System.Data.SqlServerCe.SqlCeException (0x80004005): Not enough storage is available to complete this operation.

                       at System.Data.SqlServerCe.SqlCeConnection.ProcessResults(Int32 hr)

                       at System.Data.SqlServerCe.SqlCeConnection.Open(Boolean silent)

                       at System.Data.SqlServerCe.SqlCeConnection.Open()

                       at SolarWinds.Collector.SqlCeHelper.CreateNewDatabaseConnection()

                       at SolarWinds.Collector.SqlCeHelper.ExecuteReader(SqlCeCommand command)

                       at SolarWinds.Collector.Repository.ConfigRepository.GetEntityData(DateTime version, String legacyName, Int32 batchSize)

                       at SolarWinds.Collector.DataProcessor.InventoryManager.ActualizeEntities()

                       at SolarWinds.Collector.DataProcessor.InventoryManager.ActualizeEntities(Object state)

                       at SolarWinds.Collector.Scheduler.ScheduledTask.DoWork()

                       at SolarWinds.Collector.Scheduler.ScheduledTask.RunSynch(Object state)

                    2015-01-29 19:37:03,874 [46] ERROR SolarWinds.Collector.Scheduler.ScheduledTask - ActualizeInvetoryManager threw an exception.

                    System.Data.SqlServerCe.SqlCeException (0x80004005): Not enough storage is available to complete this operation.

                       at System.Data.SqlServerCe.SqlCeConnection.ProcessResults(Int32 hr)

                       at System.Data.SqlServerCe.SqlCeConnection.Open(Boolean silent)

                       at System.Data.SqlServerCe.SqlCeConnection.Open()

                       at SolarWinds.Collector.SqlCeHelper.CreateNewDatabaseConnection()

                       at SolarWinds.Collector.SqlCeHelper.ExecuteReader(SqlCeCommand command)

                       at SolarWinds.Collector.Repository.ConfigRepository.GetEntityData(DateTime version, String legacyName, Int32 batchSize)

                       at SolarWinds.Collector.DataProcessor.InventoryManager.ActualizeEntities()

                       at SolarWinds.Collector.DataProcessor.InventoryManager.ActualizeEntities(Object state)

                       at SolarWinds.Collector.Scheduler.ScheduledTask.DoWork()

                       at SolarWinds.Collector.Scheduler.ScheduledTask.RunSynch(Object state)

                  • Re: Solarwinds Data Processor
                    irae@saccourt.ca.gov

                    Actually only applied HotFix4.  HotFix4 includes Hotfix1.