Surprising performance stats fro Orion NPM?

Question

I read somewhere that Orion NPNM running on a single polling engine can be expected to reliably monitor between 8000 and 10000 elements - regardless of the server hardware specifications.

However, I recently encountered an Orion server that seems to be violating this norm. Consider the following:

8605 Elements
957 Nodes
7104 Interfaces
544 Volumes

15821 Pollers

ICMP Polls per Sec - 30/172
SNMP Polls per Sec - 30/103

SNMP Node interval is 1 minute.

Notice that the poller is currently running at 30 polls per second for ICMP and SNMP, yet when the Polls Per Second Tuning app is run, it recommends adjusting to 172 and 103 polls per second respectively.

The CPU load and memory usage for this server are not remarkable, and all observed data graphs look normal in their granularity. So, how is it that this server is doing all of this at the default setting of 30 polls per second, especially if the SNMP node polling interval is 1 minute?

Any thoughts? Am I missing something that might explain this apparent anomoly?

patriot · Answer

Picking up on borgan's question then...the server in question is still operating at default polling intervals (120/120/120 sec  for ICMP and 10/9/15 min for SNMP).

As stated in my original post, the current polls per second are 30 for both ICMP and SNMP.

Why is Orion recommending that polls per sec be raised from 30 to 172 for ICMP and 30 to 103 for SNMP? That's a pretty significant increase for a poller that otherwise seems to be doing OK monitoring almost 9000 elements.

BTW, so how exactly does Orion ping an interface or a volume directly?

bshopp · Answer

Currently that's not the case.  We don't batch up interface and volume queries.  There's a separate query that happens for each interface and volume on the device.  Also, we state that we handle polling 8K-10K at the default polling intervals.  Based on the above comments it looks like you cranked some of those down.

borgan · Answer

Dave,

That is a good point and I believe it makes sense in light of the fact that interfaces and volumes are seen as resources of the node. Therefore, Orion receives data about those elements when it polls the node. So, it makes sense that a node with more interfaces (e.g. a switch) returns more bang for the buck (in terms of SNMP data) than a node with only one or two interfaces.

Now, can someone at SolarWinds offer an opinion on what Dave and I have said? When talking about polling performance, is it accurate to say that it matters how many interfaces per node are being polled almost as it matters how many total elements are being polled?