Hi All
I upgraded our environment to 2020.2.6HF1 and must say this was the easiest and quickest update from previous updates. Now We have 1 Main PE, 1x Web-Server and three APE's. All three PE's updated successfully but one PE's Solarwinds Log Manager for Orion Polling Service keeps stopping after about 2 seconds. Did run configuration wizard again on services and repaired the service from installer as well but still the same. Restarted server as well just to check. I don't want to do a cli uninstall and install as the hassle with moving all nodes to active again and shifting them again from main poller to remote poller is time consuming due to over 6000 objects. Any thoughts as to what I might be missing. I have logged a call with Solarwinds.
From PollingService Log File
2021-10-15 06:30:27,669 [13] WARN SolarWinds.Orion.LogMgmt.Common.Licensing.LicenseValidator - Unable to load licensed nodes, retrying #10 SolarWinds.ApiProxyFactory.ApiProxyException: Exception while acquiring API channel. ---> System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Failed to deserialize connection description of: 'OLM.BusinessLayer', 'engine:14', 'net.pipe' ---> System.InvalidCastException: Unable to cast object of type 'SolarWinds.Newtonsoft.Json.Linq.JObject' to type 'SolarWinds.ServiceDirectory.Client.Contract.IConnectionDescriptor'. at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) --- End of inner exception stack trace --- at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.<QueryAsync>b__9_0(ServiceDirectoryEntry queryResult) at System.Linq.Enumerable.<>c__DisplayClass7_0`3.<CombineSelectors>b__0(TSource x) at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext() at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<DoQueryAsync>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<GetEndpointDescriptorAsync>d__3.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) at SolarWinds.ApiProxyFactory.RootPoolInternal`2.Acquire(TServiceDescriptor serviceDescriptor) at SolarWinds.ApiProxyFactory.ApiProxy`2.GetAcquiredApiChannel() --- End of inner exception stack trace --- at SolarWinds.ApiProxyFactory.ApiProxy`2.GetAcquiredApiChannel() at System.Collections.Concurrent.ConcurrentDictionary`2.AddOrUpdate(TKey key, Func`2 addValueFactory, Func`3 updateValueFactory) at SolarWinds.ApiProxyFactory.ApiProxy`2.ApiCallInterceptor.CallWrapper.Execute() at Microsoft.Practices.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0() at Microsoft.Practices.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) at SolarWinds.ApiProxyFactory.ApiProxy`2.ApiCallInterceptor.EnsureChannelIsCachedBackOrReleased(IInvocation invocation, Action action) at SolarWinds.ApiProxyFactory.ApiProxy`2.ApiCallInterceptor.Intercept(IInvocation invocation) at Castle.DynamicProxy.AbstractInvocation.Proceed() at Castle.DynamicProxy.AbstractInvocation.Proceed() at Castle.Proxies.ILogMgmtBusinessLayerServiceProxy.IsLicenseValid() at SolarWinds.Orion.LogMgmt.Common.Licensing.LicenseValidator.ReadDataFromBusinessLayer() at SolarWinds.Orion.LogMgmt.Common.Licensing.LicenseValidator.ReadDataAndUpdateCache(CancellationToken cancellationToken) 2021-10-15 06:30:28,154 [5] WARN SolarWinds.Orion.LogMgmt.Plugin.WindowsEvents.Communication.AgentCommunication - Not running 2021-10-15 06:30:28,154 [5] WARN SolarWinds.Orion.LogMgmt.Plugin.LogFiles.Communication.AgentCommunication - Not running 2021-10-15 06:30:28,169 [13] ERROR SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient - Failed to deserialize connection description of: 'OLM.BusinessLayer', 'engine:14', 'net.pipe' System.InvalidCastException: Unable to cast object of type 'SolarWinds.Newtonsoft.Json.Linq.JObject' to type 'SolarWinds.ServiceDirectory.Client.Contract.IConnectionDescriptor'. at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) 2021-10-15 06:30:28,169 [13] ERROR ApiProxy<ILogMgmtBusinessLayerService, ServiceDirectoryQueryInternal> - Acquiring an API channel from a pool 'RootPool' failed with an exception. System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Failed to deserialize connection description of: 'OLM.BusinessLayer', 'engine:14', 'net.pipe' ---> System.InvalidCastException: Unable to cast object of type 'SolarWinds.Newtonsoft.Json.Linq.JObject' to type 'SolarWinds.ServiceDirectory.Client.Contract.IConnectionDescriptor'. at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) --- End of inner exception stack trace --- at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.<QueryAsync>b__9_0(ServiceDirectoryEntry queryResult) at System.Linq.Enumerable.<>c__DisplayClass7_0`3.<CombineSelectors>b__0(TSource x) at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext() at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<DoQueryAsync>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<GetEndpointDescriptorAsync>d__3.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) at SolarWinds.ApiProxyFactory.RootPoolInternal`2.Acquire(TServiceDescriptor serviceDescriptor) at SolarWinds.ApiProxyFactory.ApiProxy`2.GetAcquiredApiChannel() ---> (Inner Exception #0) System.InvalidOperationException: Failed to deserialize connection description of: 'OLM.BusinessLayer', 'engine:14', 'net.pipe' ---> System.InvalidCastException: Unable to cast object of type 'SolarWinds.Newtonsoft.Json.Linq.JObject' to type 'SolarWinds.ServiceDirectory.Client.Contract.IConnectionDescriptor'. at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) --- End of inner exception stack trace --- at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.<QueryAsync>b__9_0(ServiceDirectoryEntry queryResult) at System.Linq.Enumerable.<>c__DisplayClass7_0`3.<CombineSelectors>b__0(TSource x) at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext() at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<DoQueryAsync>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<GetEndpointDescriptorAsync>d__3.MoveNext()<--- 2021-10-15 06:30:28,169 [13] WARN SolarWinds.Orion.LogMgmt.Common.Licensing.LicenseValidator - Unable to load licensed nodes, retrying #11 SolarWinds.ApiProxyFactory.ApiProxyException: Exception while acquiring API channel. ---> System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Failed to deserialize connection description of: 'OLM.BusinessLayer', 'engine:14', 'net.pipe' ---> System.InvalidCastException: Unable to cast object of type 'SolarWinds.Newtonsoft.Json.Linq.JObject' to type 'SolarWinds.ServiceDirectory.Client.Contract.IConnectionDescriptor'. at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) --- End of inner exception stack trace --- at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.<QueryAsync>b__9_0(ServiceDirectoryEntry queryResult) at System.Linq.Enumerable.<>c__DisplayClass7_0`3.<CombineSelectors>b__0(TSource x) at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext() at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<DoQueryAsync>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<GetEndpointDescriptorAsync>d__3.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) at SolarWinds.ApiProxyFactory.RootPoolInternal`2.Acquire(TServiceDescriptor serviceDescriptor) at SolarWinds.ApiProxyFactory.ApiProxy`2.GetAcquiredApiChannel() --- End of inner exception stack trace --- at SolarWinds.ApiProxyFactory.ApiProxy`2.GetAcquiredApiChannel() at System.Collections.Concurrent.ConcurrentDictionary`2.AddOrUpdate(TKey key, Func`2 addValueFactory, Func`3 updateValueFactory) at SolarWinds.ApiProxyFactory.ApiProxy`2.ApiCallInterceptor.CallWrapper.Execute() at Microsoft.Practices.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0() at Microsoft.Practices.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) at SolarWinds.ApiProxyFactory.ApiProxy`2.ApiCallInterceptor.EnsureChannelIsCachedBackOrReleased(IInvocation invocation, Action action) at SolarWinds.ApiProxyFactory.ApiProxy`2.ApiCallInterceptor.Intercept(IInvocation invocation) at Castle.DynamicProxy.AbstractInvocation.Proceed() at Castle.DynamicProxy.AbstractInvocation.Proceed() at Castle.Proxies.ILogMgmtBusinessLayerServiceProxy.IsLicenseValid() at SolarWinds.Orion.LogMgmt.Common.Licensing.LicenseValidator.ReadDataFromBusinessLayer() at SolarWinds.Orion.LogMgmt.Common.Licensing.LicenseValidator.ReadDataAndUpdateCache(CancellationToken cancellationToken) 2021-10-15 06:30:28,169 [5] WARN SolarWinds.Orion.LogMgmt.Plugin.WindowsEvents.Communication.AgentCommunication - Not running 2021-10-15 06:30:28,169 [5] WARN SolarWinds.Orion.LogMgmt.Plugin.LogFiles.Communication.AgentCommunication - Not running
From BusinessLayer Log File under OLM
2021-10-15 06:27:09,533 [9] ERROR SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient - Failed to deserialize connection description of: 'OLM.BusinessLayer', 'engine:14', 'net.pipe' System.InvalidCastException: Unable to cast object of type 'SolarWinds.Newtonsoft.Json.Linq.JObject' to type 'SolarWinds.ServiceDirectory.Client.Contract.IConnectionDescriptor'. at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) 2021-10-15 06:27:09,533 [9] ERROR ApiProxy<ILogMgmtBusinessLayerService, ServiceDirectoryQueryInternal> - Acquiring an API channel from a pool 'RootPool' failed with an exception. System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Failed to deserialize connection description of: 'OLM.BusinessLayer', 'engine:14', 'net.pipe' ---> System.InvalidCastException: Unable to cast object of type 'SolarWinds.Newtonsoft.Json.Linq.JObject' to type 'SolarWinds.ServiceDirectory.Client.Contract.IConnectionDescriptor'. at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) --- End of inner exception stack trace --- at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.<QueryAsync>b__9_0(ServiceDirectoryEntry queryResult) at System.Linq.Enumerable.<>c__DisplayClass7_0`3.<CombineSelectors>b__0(TSource x) at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext() at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<DoQueryAsync>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<GetEndpointDescriptorAsync>d__3.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) at SolarWinds.ApiProxyFactory.RootPoolInternal`2.Acquire(TServiceDescriptor serviceDescriptor) at SolarWinds.ApiProxyFactory.ApiProxy`2.GetAcquiredApiChannel() ---> (Inner Exception #0) System.InvalidOperationException: Failed to deserialize connection description of: 'OLM.BusinessLayer', 'engine:14', 'net.pipe' ---> System.InvalidCastException: Unable to cast object of type 'SolarWinds.Newtonsoft.Json.Linq.JObject' to type 'SolarWinds.ServiceDirectory.Client.Contract.IConnectionDescriptor'. at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) --- End of inner exception stack trace --- at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.<QueryAsync>b__9_0(ServiceDirectoryEntry queryResult) at System.Linq.Enumerable.<>c__DisplayClass7_0`3.<CombineSelectors>b__0(TSource x) at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext() at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<DoQueryAsync>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<GetEndpointDescriptorAsync>d__3.MoveNext()<--- 2021-10-15 06:27:09,533 [9] WARN SolarWinds.Orion.LogMgmt.Common.Licensing.LicenseValidator - Unable to load licensed nodes, retrying #44878 SolarWinds.ApiProxyFactory.ApiProxyException: Exception while acquiring API channel. ---> System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Failed to deserialize connection description of: 'OLM.BusinessLayer', 'engine:14', 'net.pipe' ---> System.InvalidCastException: Unable to cast object of type 'SolarWinds.Newtonsoft.Json.Linq.JObject' to type 'SolarWinds.ServiceDirectory.Client.Contract.IConnectionDescriptor'. at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) --- End of inner exception stack trace --- at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.FromProperties(ServiceDirectoryEntry queryResult) at SolarWinds.ServiceDirectory.Client.ServiceDirectoryClient.<QueryAsync>b__9_0(ServiceDirectoryEntry queryResult) at System.Linq.Enumerable.<>c__DisplayClass7_0`3.<CombineSelectors>b__0(TSource x) at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext() at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<DoQueryAsync>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at SolarWinds.ApiProxyFactory.ServiceDirectory.EndpointDescriptorProvider`1.<GetEndpointDescriptorAsync>d__3.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) at SolarWinds.ApiProxyFactory.RootPoolInternal`2.Acquire(TServiceDescriptor serviceDescriptor) at SolarWinds.ApiProxyFactory.ApiProxy`2.GetAcquiredApiChannel() --- End of inner exception stack trace --- at SolarWinds.ApiProxyFactory.ApiProxy`2.GetAcquiredApiChannel() at System.Collections.Concurrent.ConcurrentDictionary`2.AddOrUpdate(TKey key, Func`2 addValueFactory, Func`3 updateValueFactory) at SolarWinds.ApiProxyFactory.ApiProxy`2.ApiCallInterceptor.CallWrapper.Execute() at Microsoft.Practices.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0() at Microsoft.Practices.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) at SolarWinds.ApiProxyFactory.ApiProxy`2.ApiCallInterceptor.EnsureChannelIsCachedBackOrReleased(IInvocation invocation, Action action) at SolarWinds.ApiProxyFactory.ApiProxy`2.ApiCallInterceptor.Intercept(IInvocation invocation) at Castle.DynamicProxy.AbstractInvocation.Proceed() at Castle.DynamicProxy.AbstractInvocation.Proceed() at Castle.Proxies.ILogMgmtBusinessLayerServiceProxy.IsLicenseValid() at SolarWinds.Orion.LogMgmt.Common.Licensing.LicenseValidator.ReadDataFromBusinessLayer() at SolarWinds.Orion.LogMgmt.Common.Licensing.LicenseValidator.ReadDataAndUpdateCache(CancellationToken cancellationToken)
