"SNMP Monitor Execution Error. Timeout" when using snmpv3

Hello,

We are using the following: Orion Core 2011.2.2, SAM 5.0.0, NPM 10.2.2, IVIM 1.2.0. I am using a test server running REHL 6 that is configured identically to some of our production servers, but is not joined to our AD environment. I was hoping to move to SNMPv3, but after creating an SNMPv3 user for Orion to connect with, I am getting an issue.

All polling seems to work - when adding the node to Orion I see all the details/resources I expect to see. However, when trying to monitor a particular running process, immediately after polling I get the error message "SNMP Monitor Execution Error. Timeout". This process monitor works fine in SNMPv2. The application monitor is set to the default timeout of 300 seconds, yet this error occurs immediately. I have no idea why. Whether I use credentials that were set up as an SNMPv3 user, or root credentials, I cannot get the application monitor to run successfully using SNMPv3.

On a side note, I have a few production servers that were rebuilt. The same version of SNMP is installed on these RHEL servers as is installed on my test server. Identical packages are installed. There are currently no firewall restrictions in place. The snmpd.conf file used for SNMPv2 is identical to the one I use on the test server (which works fine) and is the same conf file that was used on these servers prior to being rebuilt. When the servers were being rebuilt, the nodes were removed from Orion. Adding the nodes back in after rebuild, when running an SNMP test, the results come back with a PASS. However, when I hit "next" and Orion starts to poll the server to see what resources are available, it comes up empty, with nothing available for monitoring. This happens with 3 machines. I cannot understand how an identical RHEL + SNMP installation using an identical snmpd.conf file will work on one machine but not others.

Any advice on where to look would be appreciated, for resolving either matter above.

Find more posts tagged with

Accepted answers

All comments

aLTeReGo

In most cases the issue with SNMPv3 is related to the views allowed in the snmpd.conf file. Essentially the problem is that the SNMP user as defined is only allowed access to certain MIB/OIDs, and not other. ".1" should always be used as the included view.

E.G. "view all included .1"

FormerMember

Thanks for your reply.

According to the Red Hat documentation on SNMPv3 configuration, after creating a v3 user and specifying the encryption and authentication protocols, modifying the snmpd.conf file as follows would be enough to grant access to all available MIBs:

directive user [noauth|auth|priv] [OID]

"rwuser admin authpriv .1"

I have a read-only user (rouser) set up in just this fashion. Using an external MIB-Browser, with these credentials I am able to walk the entire tree on the server in question, and locate the process I am trying to monitor from the HOST-RESOURCES-MIB-rfc2790. I first set up this process monitor by doing the following:

SAM Settings>Find Processes, Services...>Select component monitor type - Process Monitor SNMP>Select a target - browse to server>Check onr or more processes...

At this point, Orion SAM appears to load a list of all processes running on the indicated server, including the process which I wish to monitor - it identifies an instance count, with some process running multiple instances, telling me it is actually pulling this information from the server, not just from a generic entry in a MIB. When I select the process I wish to monitor and complete the procedure, Orion SAM attempts to poll the process and delivers the timeout message mentioned above.

I have tried other processes to monitor in this way as well - all can be selected from the list of processes, but all fail when trying to poll. This poller works in SNMPv2.

The fact that I can use the SNMPv3 user/credentials that I created to walk the entire tree of the server using other methods indicates that SNMPv3 is configured correctly. The snmpd.conf man page at http://linux.die.net/man/5/snmpd.conf shows an example specifying either a view or an OID. Is it absolutely necessary to configure/specify a view? Everything else appears to function properly by just specifying the OID.

I have run snmpd logging debug information but don't see anything indicating any errors...

Any further speculation on where to look would be helpful...

aLTeReGo

Have you tried gathering a MIB walk of the device using SNMPv2 and comparing that to SNMPv3? The number of lines returned in both samples should be identical. If the SNMPv3 version comes up short it's like a view issue.

FormerMember

Hmmm, after doing a MIB walk with the software you provided, v3 stops at 840 OIDs with the message "ASN.1 Decoding Error - Packet was unable to be decoded at the agent." (v2 walk produced 12,920 lines). Since it was able to scan at least 840 OIDs I'd guess the SNMPv3 configuration is not the issue, but how do I go about troubleshooting this decoding error - any suggestions? Google was not entirely forthcoming...

FormerMember

After searching thwack, I noted that a user had used the mib walking tool in the engineers toolset to successfully get results after experiencing an error similar to mine, so I downloaded the trial and ran the MIB Walk tool that comes with the toolset - unfortunate I also received an error there - "Error during Walk: An attempt was made to load a program with an incorrect format. Extra Info: Error Decoding packet."

EDIT: After attempting the walk with the tool in the engineer's toolset, it fails at different times. It displays a message "Walking MIB tree..." and a running number count - after multiple tests I have seen it fail as soon as 100 and run as long as 1069 - I don't know if the walk reads through the information base randomly or reads the tree top down - I would think top down - which seems odd that it is apparently failing randomly... multiple runs of the MIB Walk utility you included in your reply to me fail constantly at 840 - this is after closing the tool and re-opening it and running it fresh...

EDIT #2: When using a different 3rd party tool to run a MIB Walk (HiliSoft MIB Browser), I can do a walk using v3 credentials and get 12,943 lines.....

Message was edited by: Steve Duff

Petr.Vilem

Could you try disabling bulk loading of process information? There is difference between wizard (Find Processes, Services...) and regular polling: wizzard is using simple GetNext, but behavior of regular probe obeys configuration. There is kb for disabling bulk for snmp process: SolarWinds Knowledge Base :: Monitoring or Polling of your Cisco Call Manager stops working after upgrading to SAM 5.0. and kb that describes this configuration section in more details: SolarWinds Knowledge Base :: Problems in Orion APM with SNMP Process Monitors failing

Another difference between wizard and regular polling is, that wizard is loading list of process only from one table (HOST-RESOURCES-MIB:hrSWRunTable 1.3.6.1.2.1.25.4.2 - list of processes), but performance data are loaded from additional table (HOST-RESOURCES-MIB:hrSWRunPerfTable 1.3.6.1.2.1.25.5.1) during regular poll.

FormerMember

Disabling of bulk loading of process information did not change anything. I have not tried using 3DES yet, but am now faced with another issue of more importance. I have posted elsewhere on thwack with regards to Advanced Alert Manager being broken. Until I get this issue resolved I am not concerned with the process monitor not working in snmp v3