Hello,
Since installation of HF2 we are facing a memory leak with SolarWinds Cortex service on Primary Pooler. It is eating up memory till there is no more available memory on the server and application crashes, we are then forced to either restart the Cortex service or whole server:
I tried the run the configuration wizard and repair the service from control panel but without luck.
We are basically forced to restart it every 3-4 hours, that's how fast the memory is depleted (and there is 40 GB assigned memory to the server)
I went through Cortex log and noticed that service is spamming so much log entries that it's creating a new log file every two minutes:
There are two kinds of entries that are repeated almost indefinitely:
2021-01-17 10:39:44,744 [4] ERROR RestApi.SolarWinds.Cortex.Internal.Commands.CommandExecutor - Error executing command 'Core.AddToCortex' on element type 'Orion.Node'. System.ArgumentException: Element type 'Orion.Node' does not exist.
at SolarWinds.Cortex.Internal.Commands.CommandExecutor.GetElementType(String elementTypeName)
at SolarWinds.Cortex.Internal.Commands.CommandExecutor.<>c__DisplayClass7_0.<ExecuteCommandAsync>g__GetElementTypeAndInstance|0()
at async SolarWinds.Cortex.Internal.Commands.CommandExecutor.ExecuteCommandInternalAsync(?)
and
2021-01-17 10:35:13,317 [5] INFO SolarWinds.InstanceProxyGenerator.ProxyCommunicationChannelWrapper - Processing incomming message has failed. The sender should retry the operation.
SolarWinds.ServiceHost.Internal.Exceptions.ServiceHostManagerCommunicationException: 6fb1d71e-cd67-4b92-b8c8-0b205f6408d2: Unable to send agent message SendProxyPacketAsync ---> SolarWinds.ServiceHost.Internal.Exceptions.ServiceHostManagerCommunicationException: 6fb1d71e-cd67-4b92-b8c8-0b205f6408d2: Response to message 'SendProxyPacketAsync' was not received in '00:02:00'. Response: ID: 16e832a3-6cc5-4acb-a618-da0d193f6ce9 Source: 94D5DBAD-5E75-437B-A588-8DE667E87C23 Sender: Cortex Target: Recipient: ServiceHostManager Type: SendProxyPacketAsync Timeout: 00:02:00 RemainingTimeout: 00:01:59.9829976 CreationTime: 1/17/2021 9:35:13 AM, Files: 0 ResponseStatus: DeliveryTimeout ResponseTo: 87c1588a-e032-4552-9642-fca333b60fc3 Data:
at SolarWinds.ServiceHost.AgentSupport.MessagingServiceBase`1.ProcessResponse[TResponse](Guid traceId, UniversalMessage response, TMessage message)
at SolarWinds.ServiceHost.AgentSupport.MessagingServiceBase`1.<SendMessageAndGetResponseAsync>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SolarWinds.ServiceHost.AgentSupport.MessagingServiceBase`1.<SendMessageAndWaitForConfirmationAsync>d__9.MoveNext()
--- End of inner exception stack trace ---
at SolarWinds.ServiceHost.AgentSupport.MessagingServiceBase`1.<SendMessageAndWaitForConfirmationAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SolarWinds.ServiceHost.AgentSupport.ServiceHostManagerAgentChannel.<Send>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SolarWinds.ServiceHost.Internal.ProxyCommunicationChannel.<SendAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at SolarWinds.ServiceHost.Internal.ProxyCommunicationChannel.<SendAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SolarWinds.InstanceProxyGenerator.ProxyCommunicationChannelWrapper.<ProcessIncomingCall>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SolarWinds.InstanceProxyGenerator.ProxyCommunicationChannelWrapper.<OnChannelDataReceivedAsync>d__18.MoveNext()
Anyone encountered anything similar or can suggest what else can be done to get this fixed?
I have a ticked opened with support (00706884) but I may get some useful responses from here as well.
