Hello,
We are monitoring over 1200 remote nodes via WMI, but get the same results using RPC. The Nodes are Windows 7 Workstation and Windows 10 Workstation.
At any one time at least 30 nodes are displaying the error "One or more application monitors is in an Unknown state."
On each node, we are monitoring between 6 and 8 applications, with 1 up to all of the application monitors in this state.
This is no visible pattern, with occurrences ignoring APE, OS, or Application.
The nodes are split between 2 APE's
The polling rates for the 2 APE are less than 30%, with SAM polling rates around 20%
Polling completion is usually100%
Application/Component Polling frequencies are between 5 minutes and 30 minutes.
Currently running
SAM 6.3.0
NPM 12.0.1
We are attempting ServiceNow integration, but the "flood" of unknown events is causing issues for our new SN admin.
All previous research on the issue leads seems to lean towards a static issue, and as I see it, does not apply to the intermittent issues we are seeing.
In attempts to determine the cause, I enabled debugging for a time, which provided the following common errors. Theses errors are then followed by
2017-05-27 15:22:47,353 [STP Pool:2635 Thread #0] [C0] ERROR SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor - Error in ExecuteGroup occured.
System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
at System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(Int32 errorCode, IntPtr errorInfo)
at System.Management.ManagementObjectSearcher.Get()
at SolarWinds.APM.Probes.ProcessInfo.ProcessAllProcesses(Dictionary`2 processes, ManagementObjectSearcher searcher, Action`2 proc)
at SolarWinds.APM.Probes.ProcessInfo.GetPerfCounterDeltaInFiveSeconds(ManagementScopeContext scope, Dictionary`2 processes, EnumerationOptions options)
at SolarWinds.APM.Probes.ProcessInfo.GetProcessInfo(ManagementScopeContext scope, IEnumerable`1 pids, IList`1 processNames, IList`1 commandLineFilters)
at SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor.GetProcessInfoViaWmi(String ipAddress, UserNamePasswordCredential credential, List`1 services)
at SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor.ExecuteGroup(List`1 serviceProbeInfos)
2017-05-27 15:22:47,369 [STP SmartThreadPool Thread #178] [C(null)] ERROR SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor - Unhandled exception.
System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
at System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(Int32 errorCode, IntPtr errorInfo)
at System.Management.ManagementObjectSearcher.Get()
at SolarWinds.APM.Probes.ProcessInfo.ProcessAllProcesses(Dictionary`2 processes, ManagementObjectSearcher searcher, Action`2 proc)
at SolarWinds.APM.Probes.ProcessInfo.GetPerfCounterDeltaInFiveSeconds(ManagementScopeContext scope, Dictionary`2 processes, EnumerationOptions options)
at SolarWinds.APM.Probes.ProcessInfo.GetProcessInfo(ManagementScopeContext scope, IEnumerable`1 pids, IList`1 processNames, IList`1 commandLineFilters)
at SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor.GetProcessInfoViaWmi(String ipAddress, UserNamePasswordCredential credential, List`1 services)
at SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor.ExecuteGroup(List`1 serviceProbeInfos)
I am hoping somebody has experienced similar issues and can shed some insight on next steps we can pursue.
