Error in alert "Unexpected Error Occurred. Out of memory"

We've been experiencing a strange error with one of our monitored applications.

Our alert rule states to fire an email when the state of this application is NOT "Up". So an alert email notification will trigger even if the monitor is in an "Unknown" state.

Inside the alert rule we include the component status message. In this instance we see the following in the alert notification:

The monitor on NODE05 is Unknown for the following reason:

Unexpected error occurred. Out of memory

Alert was triggered at Monday, December 5, 2016 9:54 PM.

A few minutes later the alert resets and the APM monitor works normally for several hours before this happens again.

I found the APM logs for this particular application and have these entries:

2016-12-05 21:49:12,721 [STP Pool:184 Thread #0] [A4526] [C0] ERROR SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor - Error in ExecuteGroup occured.

System.Management.ManagementException: Out of memory

at System.Management.ManagementException.ThrowWithExtendedInfo(ManagementStatus errorCode)

at System.Management.ManagementObjectCollection.ManagementObjectEnumerator.MoveNext()

at SolarWinds.APM.Probes.ProcessInfo.ProcessAllProcesses(Dictionary`2 processes, ManagementObjectSearcher searcher, Action`2 proc)

at SolarWinds.APM.Probes.ProcessInfo.GetPerfCounterDeltaInFiveSeconds(ManagementScopeContext scope, Dictionary`2 processes, EnumerationOptions options)

at SolarWinds.APM.Probes.ProcessInfo.GetProcessInfo(ManagementScopeContext scope, IEnumerable`1 pids, IList`1 processNames, IList`1 commandLineFilters)

at SolarWinds.APM.Probes.ProcessInfo.GetProcessInfo(ManagementScopeContext scope, IEnumerable`1 pids)

at SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor.GetProcessInfoViaWmi(String ipAddress, UserNamePasswordCredential credential, List`1 services)

at SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor.ExecuteGroup(List`1 serviceProbeInfos)

2016-12-05 21:49:12,737 [STP SmartThreadPool Thread #38] [A4526] [C(null)] ERROR SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor - ManagementException caught. MamagementStatus code: OutOfMemory

System.Management.ManagementException: Out of memory

at System.Management.ManagementException.ThrowWithExtendedInfo(ManagementStatus errorCode)

at System.Management.ManagementObjectCollection.ManagementObjectEnumerator.MoveNext()

at SolarWinds.APM.Probes.ProcessInfo.ProcessAllProcesses(Dictionary`2 processes, ManagementObjectSearcher searcher, Action`2 proc)

at SolarWinds.APM.Probes.ProcessInfo.GetPerfCounterDeltaInFiveSeconds(ManagementScopeContext scope, Dictionary`2 processes, EnumerationOptions options)

at SolarWinds.APM.Probes.ProcessInfo.GetProcessInfo(ManagementScopeContext scope, IEnumerable`1 pids, IList`1 processNames, IList`1 commandLineFilters)

at SolarWinds.APM.Probes.ProcessInfo.GetProcessInfo(ManagementScopeContext scope, IEnumerable`1 pids)

at SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor.GetProcessInfoViaWmi(String ipAddress, UserNamePasswordCredential credential, List`1 services)

at SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor.ExecuteGroup(List`1 serviceProbeInfos)

2016-12-05 21:49:12,737 [STP SmartThreadPool Thread #38] [A4526] [C(null)] ERROR SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor - ManagementException caught. MamagementStatus code: OutOfMemory

System.Management.ManagementException: Out of memory

at System.Management.ManagementException.ThrowWithExtendedInfo(ManagementStatus errorCode)

at System.Management.ManagementObjectCollection.ManagementObjectEnumerator.MoveNext()

at SolarWinds.APM.Probes.ProcessInfo.ProcessAllProcesses(Dictionary`2 processes, ManagementObjectSearcher searcher, Action`2 proc)

at SolarWinds.APM.Probes.ProcessInfo.GetPerfCounterDeltaInFiveSeconds(ManagementScopeContext scope, Dictionary`2 processes, EnumerationOptions options)

at SolarWinds.APM.Probes.ProcessInfo.GetProcessInfo(ManagementScopeContext scope, IEnumerable`1 pids, IList`1 processNames, IList`1 commandLineFilters)

at SolarWinds.APM.Probes.ProcessInfo.GetProcessInfo(ManagementScopeContext scope, IEnumerable`1 pids)

at SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor.GetProcessInfoViaWmi(String ipAddress, UserNamePasswordCredential credential, List`1 services)

at SolarWinds.APM.Probes.ProbeExecutors.WmiWindowsServiceBatchExecutor.ExecuteGroup(List`1 serviceProbeInfos)

What concerns me is the entries that say "ManagementException caught. ManagementStatus code: OutOfMemory".

Would this indicate a problem with the Orion APM or the monitored Node?

Find more posts tagged with

application error

sam 6.3

apm

Accepted answers

All comments

I went ahead and opened up support case number # 1089448 for this.

Check the memory configuration on the Solarwinds Server. WMI is CPU but also memory intensive; Admin guide states for every 1000 components you should allocate an additional 5GB of RAM.

Utilizing the Windows and Linux Agents can reduce the need for more resources on the Solarwinds Server as the resources on the remote systems will be used.

Did you ever get a response to your support case? I'm seeing the same issue.

Resurrecting this thread...

So I'm seeing this too, but I am using the Windows agent. Memory utilization on the agent server is in the 65% range. Application monitor (and the template) is set to use Agent polling by default.

Was there any update to this?

I am also seeing this issue with both agent and non-agent nodes. The strange thing about it is my Polling Engine has plenty of unused memory.

This error is typically returned by the host being monitored and is not the Orion server itself which has run out of memory. By default, the memory quota limit for instances of wmiprvse.exe on Windows XP and Windows Server 2003 is 128 MB, and 512 MB on newer Operating Systems (Vista and higher).

You may have encountered a memory leak in WMI as there have been several fixes released by Microsoft over the years for this type of behavior. The 'fix' would be applied to the nodes being monitored by Orion.

Awesome, thanks for the info aLTeReGo!

Sorry to resurrect this but I am getting the same error.

The odd thing is we are running Windows agents on these boxes so why would it be polling with WMI? Same as joe.tran, the Application monitor (and the template) are set to use Agent polling.

WMI is simply an API. One which can be remoted as most APIs can. The Agent does make use of WMI, as well as other Windows based APIs.

Ahh okay. That makes sense. Thank you aLTeReGo!

My issue might be slightly different then. I am running windows service monitor components on multiple nodes and every once in awhile, this one will go into an "Unknown" state for one poll and then back to normal for the next poll. A couple of the error messages that I've seen while it is in an "Unknown" state are:

- Unexpected error occurred. Out of memory

- Unexpected error occurred. Provider failure

- Unexpected error occurred

I have the same template running on multiple Nodes but it appears this issue is only occurring on one of them, which makes me think the issue might be with the Node itself and not with Solarwinds or the template...

WMI has conservative default limits on the amount of memory it is allowed to utilize for fulfilling queries. From the error, it suggests that you're running into this limit on occasion. I would suggest increasing this limit and seeing if that resolves the issue. Note that the limit is a fixed number of bytes rather than a percentage of total RAM, so the limit has been designed to protect systems memory exhaustion from WMI on machines with a total of 512MB of RAM. This is much less of an issue for today's systems which typically have at least 4GB of RAM.

Reviving ticket again.

I'm having this issue as well. The extra bit of detail I have in my situation is that I see a CPU Spike on my poller at the time my SAM check returns a 'Unknown' status. The issues clear once the polling engine is back under moderate CPU load.

Could this issue be more of a resource issue on the poller side not the agent/Node side?

I would definitely check both, but if there's correlation it could be a cause.

Do you see anything in the logs (specifically core business layer, business layer host) on the poller? Are there any windows event logs like a service is crashing when the spike happens?

It's only a theory, but as many of the services are 32 bit, but can spin off additional processes, I wonder if one of them is hitting their ~1.8-2 GB limit and crashing before recovering.

A lot more information would be needed to know for sure, and be sure to check the agent logs (if you're using the agent) on the node as well, especially if it's only one node having the issue.

Quick Links