cancel
Showing results for 
Search instead for 
Did you mean: 
Level 10

Errors in Event Log from servers with > 32 logical CPUs

Jump to solution

I have been noticing a lot of EventID 2006 entries in the Application Event Logs of our 40 core Dell R910 servers.  These machines run Windows 2008 R2 Enterprise SP1.

"Unable to read Server Queue performance data from the Server service. The first four bytes (DWORD) of the Data section contains the status code, the second four bytes contains the IOSB.Status and the next four bytes contains the IOSB.Information."

I believe these are coming from the Orion polling engine, as it seems to be a 32-bit process.  This KB article from Microsoft seems to explain what I'm dealing with: 32-bit application cannot query performance "Server Work Queues" counters on Windows Server 2008 R2-...

My guess would be that Orion encounters this issue when doing the individual CPU core polling for the machine view in Managed Nodes.  For these systems, SAM only finds data for the first 32 cores.  I thought at first that it was some limitation of the chart type and forgot about it a long time ago, but now I'm not so sure.

Is this a known issue with Orion SAM and many-core machines? 

1 Solution

Accepted Solutions
Highlighted
Level 12

Re: Errors in Event Log from servers with > 32 logical CPUs

Jump to solution

Hello,

Just one note: The node properties like CPU, memory, volumes, interfaces, hardware health information, asset inventory information are polled through 32-bit process. You can switch to use 64-bit process only for SAM applications. We will consider the possibility to use 64-bit process also for node properties (it is tracked internally under FB275825).

Lukas Belza (SolarWinds development)

View solution in original post

44 Replies
Highlighted
Level 17

Re: Errors in Event Log from servers with > 32 logical CPUs

Jump to solution

This does appear to be an issue with WMI-based polling and >32 CPUs. Unfortunately I don't have a 40-core box to test with off hand- if you switch to SNMP polling, are we able to to return data for all CPUs?

Highlighted
Level 10

Re: Errors in Event Log from servers with > 32 logical CPUs

Jump to solution

Good idea.  I have another 40-core server that was using SNMP already, but I hadn't thought to check.  Indeed I do see data on all 40 cores for it in Manage Nodes.  Strange though, that server also has all the 2006 entries filling up the App Event Log.  Maybe they're coming from somewhere else after all?  

Next I will unmanage the node and see if the messages are still logged on that machine.  If they are, then I guess I can't blame SAM.

0 Kudos
Highlighted
Product Manager
Product Manager

Re: Errors in Event Log from servers with > 32 logical CPUs

Jump to solution

It may be possible that these EventIDs are being generated by Windows Service or Process component monitors in SAM. You may want to try changing the applications assigned to that node to use the 64bit Job Engine and see if the Windows Events no longer continue to appear. Another option might be to change the fetching method for those process and service monitors to use RPC instead of WMI.

64bit.png

RPC.png

Highlighted
Level 10

Re: Errors in Event Log from servers with > 32 logical CPUs

Jump to solution

Hey, interesting.  I knew about the 32/64bit polling option a long time ago when writing a bunch of PowerShell components, but forgot all about it.    I'll give that a shot.

0 Kudos
Highlighted
Level 12

Re: Errors in Event Log from servers with > 32 logical CPUs

Jump to solution

Hello,

Just one note: The node properties like CPU, memory, volumes, interfaces, hardware health information, asset inventory information are polled through 32-bit process. You can switch to use 64-bit process only for SAM applications. We will consider the possibility to use 64-bit process also for node properties (it is tracked internally under FB275825).

Lukas Belza (SolarWinds development)

View solution in original post

Highlighted
Level 10

Re: Errors in Event Log from servers with > 32 logical CPUs

Jump to solution

Thank you for the information.  My test of unmanaging the node did prove that the events were coming from SAM polling.  I also ran the same test with our Nagios server and found that Nagios's check_nt plugin causes the same issue, so it's not like this is uncommon.

0 Kudos
Highlighted
Level 8

Re: Errors in Event Log from servers with > 32 logical CPUs

Jump to solution

I see this post is a year old now but is it still the case that you cannot upgrade the node properties like CPU, memory, volumes, interfaces, hardware health information, asset inventory information to 64-bit?

I'm having the same problem as the original poster but with just the basic node properties from NPM.  The host that is being polled has 40 logical processors so the 32-bit process is returning the above error every 5 minutes.

MS released the following patch for the 64-bit processes if you have more then 64 logical processors but I don't see one for 32-bit. http://support.microsoft.com/kb/2733563

0 Kudos
Highlighted

Re: Errors in Event Log from servers with > 32 logical CPUs

Jump to solution

I am seeing a lot of these as well. Wondering what a fix might be?

0 Kudos
Highlighted
Product Manager
Product Manager

Re: Errors in Event Log from servers with > 32 logical CPUs

Jump to solution

You may consider changing the node polling method from WMI to SNMP for nodes that exhibit this behavior.

0 Kudos