2 Replies Latest reply on Mar 1, 2016 5:37 PM by michael.rohrbacher

    Issues after upgrade to NCM 7.4 & Latest Hotfix

    michael.rohrbacher

      This is probably coincidence and unrelated, but we have noticed after upgrading to NCM 7.4 and applying the latest hotfix for NPM, we are now having problems with the polling engine stopping randomly and also with the alerting engine service not working properly due to intermittent database connections. This has happened 4 times since the upgrade (2 weeks ago) and we have not had any problems like this in the past. I'm not saying these problems are caused by Orion, just looking for some info. Here is what the logs look like.

       

      First Event

       

      Service was unable to open new database connection when requested.

       

      SqlException: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The semaphore timeout period has expired.)

       

      Connection string - Data Source=XXXXXXXX ;Initial Catalog=SolarWindsOrion;Persist Security Info=False;User ID=XXXXXX ;Password=*******;Connect Timeout=20;Load Balance Timeout=120;Packet Size=4096;Application Name=SolarWinds.BusinessLayerHost@domain-SolarWinds.UDT.BusinessLayer.dll;Workstation ID=SolarWinds

       

      Second Event

      SolarWinds.Orion.Core.BusinessLayer - Unhandled exception: Exception of type 'System.OutOfMemoryException' was thrown.

       

      This is followed by all Orion services dumping and restarting

       

      Starting Core Service [XXXXXXX]

       

      We are also now seeing messages about the alerting engine such as:

       

      2016-02-28 04:43:54,289 [MainTaskThread] ERROR All - Error in SetupDBConnection System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out

       

         at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)

       

         at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)

       

         at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)

       

         at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()

       

         at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()

       

         at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()

       

         at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)

       

         at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)

       

         at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()

       

         at System.Data.SqlClient.SqlDataReader.get_MetaData()

       

         at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)

       

         at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)

       

         at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)

       

         at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)

       

         at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)

       

         at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)

       

         at AlertingEngine.SWAlertingEngine.SetupDBConnection(SqlConnection& DBConnection, SqlCommand& DBCommand, SqlDataReader& DBReader)

       

      ClientConnectionId:11fce9e4-87b4-43c8-a056-aa6acf872c57

       

      Error Number:-2,State:0,Class:11

       

       

       

      Orion Platform 2015.1.2, NCM 7.4, IVIM 2.1.0, QoE 2.0, NPM 11.5.2, NTA 4.1.1, UDT 3.2.2