5 Replies Latest reply on Jan 29, 2011 4:22 PM by Paul_H

    Is there a better way to do Process monitoring via SNMP

      We monitor processes via SNMP using a two line powershell script which does a snmpwalk of the current running processes on the target  machines host_resource mib, returning a 0 or 1, depending it it finds the process or not. This simple script works for both Unix and Windows successfully however we have hit two problems.

      The first, is that it cannot differentiate multiple processes by the same name. 

      The second problem is that the overhead of using powershell for these 20 external process monitors can be greater than ipmonitor itself which has over 3000 active standard monitors. I think this is because powershell is using CPU to actively check if the snmpwalk has come back yet, which can take a second or two.

      Below is the simple but effective generic script which we pass the target server, community string and process name as parameters. It runs on the IPMONITOR host machine, requiring nothing installed on the target servers.

      if (invoke-expression "d:\net-snmp\usr\bin\snmpwalk -v 1 -c $($args[0]) $($args[1]) 1.3.6.1.2.1.25.4.2.1.2" | select-string "$($args[2])") { Write-Host "FOUND"; exit 1 } else { Write-Host "NOT FOUND";exit 0 }

      Does anyone have a better/faster/more effecive/alternate way of being able to solve or improve either of these issues?

      Does anyone know if process monitoring, including monitoring multiple processes of the same name, will be added as a monitor type at any point in IPMONITOR's future?

      Paul

        • Re: Is there a better way to do Process monitoring via SNMP
          Fodome

          Hello Paul,

          The following VBScript can be used against Windows systems which will likely use less overhead:

          Monitoring the Existence of a Specific Process on a Remote Machine Running Windows 2000/XP/2003

          Monitoring a process on a UNIX system can be done using the Custom SNMP Monitor and some minor modifications to the snmpd.conf file on the remote system.  More on this is found here:

          http://support.ipmonitor.com/tutorials/f699eb0da63948a6b906f66711f4c62f.aspx

          Neither of these methods can differentiate between processes with the same name.  How would you even do that?

          Sincerely,

          Chris Foley - SolarWinds - Support Specialist
          Support:866.530.8040 || Fax:512.857.0125
          network management simplified  |  solarwinds.com

            • Re: Is there a better way to do Process monitoring via SNMP

              Hi Chris,

              Thanks for the quick reply!!,

              The VBScript you pointed to uses WMI which is unfortunate for me. Is there a SNMP version of this?

              As best as I understand, to differentiate with the same name, you need to find the OID of the processes you want to monitor and then walk the command paramaters of the OIDS for the running processes and match them up based on process number(I manually did this to verify my understanding). Way beyond my scripting skills to achieve as the process number changes each time a process stops and starts, adding to the complexity. ORION APM process monitor (both SNMP and WMI) does have the option to provide a command line filter which allows for unique identification multiple proccesses of the same name - you just need to know them first.

              Our worst case scenario is to have CA-Spectrum, our trap receiver, do process monitoring as it also does SNMP unique process monitoring via RFC2790 SNMP polling and internally correlating the command line to the process name. It presents the processes as a table which you tick box the processes you want to monitor, with the command line already displayed, making selection quite easy.

              ***Update - CA-SystemEDGE can also differentiate processes of the same name via SNMP.***

              ***WhatsUp Gold has native process monitoring using SNMP or WMI. WMI also gives CPU and Memory whereas SNMP gives up/down only***

              Strangely enough, the requirement for this is for two reasons. The first, multiple java.exe processes was obvious. The second requirement came from the same application team that have an application that will restart itself in default mode, rather than customised mode, if it has an error. The only way to tell is by monitoring the command parameters it is started with. 

              IPMONITOR is of course my prefered solution as I can get the current status and history of the monitor, as well as have in one screen, an overall view of all of my application status.

              Paul

                • Re: Is there a better way to do Process monitoring via SNMP

                  OK...some thoughts on how it is might be done:

                  The host resource running processes OID is read on the target host. OID is 1.3.6.1.2.1.25.4.2.1.2 This gives all the current running processes.

                  If you installed net-snmp on the target server, Net-snmp proc extention does the reading, having already been installed on the target server, updating a net-snmp OID with a value stating if there are zero or more than zero processes running.

                  If you did not install net-snmp on every server and do not want to have to customise snmp on a per-server basis, you have to do something similar to the script at the top, walking the running process OID and scanning for the process you are looking for.

                  However, if you want to check for unique processes, you need to also walk the host resource process running paramters OID 1.3.6.1.2.1.25.4.2.1.5 which you then match up with the running process table based on the process id which is the last part of the OID. See below for an example of processes 1848 and 1912.

                  .1.3.6.1.2.1.25.4.2.1.2.1848 = STRING: "stSchedEx.exe"
                  .1.3.6.1.2.1.25.4.2.1.2.1912 = STRING: "svchost.exe"

                  .1.3.6.1.2.1.25.4.2.1.5.1848 = STRING: "/Service"
                  .1.3.6.1.2.1.25.4.2.1.5.1912 = STRING: "-k termsvcs"

                  Now, how would you build a monitor for this that runs on the IPMONITOR host only?

                  1/    You could do a walk of both OID's, build them into an array, and then do a look up of the combined process name and paramters to find the unique match.

                  2/    You could seperate out the array building from the checking. Once you have done the initial array build, you know the process id and more importantly, the unique OID of the unique process you wish to monitor. As Windows increments the PID whenever one stops and starts, monitoring the OID with the processid you want is now an easily implemented, low overhead,  standard IPMONITOR SNMP User Experience monitor. 

                  Of the two options, the first takes more overhead as you building the array each time and also doing snmpwalk's of two potentially very large OID's, which can take excessively long.

                  The second option makes the checking whilst the process is up very easy and efficient - a simple 1 OID SNMP GET.  The challenge, is how to have IPMONITOR allow for a variable OID in the OID field as PID, the last part of the OID, changes every time the process restarts. My thinking got as close as using the monitor down alert to trigger an external process which could do the array read, but there is no way to get the new OID back into the OID field of the monitor within IPMONITOR. There is also the difficulty in how long for, how often, and by who, the external process runs.

                  I could use the extenal process to trigger a Windows Scheduler task to run every x minutes and once it had found the process and paramters that I wanted, it updates IPMONITOR and disables itself.  I could also have an external process monitor that is disabled/suspended and that it is triggered to enabled when the sNMP get failes. The advantage of this is that it is in the same group and display as the process monitor, runs only when the first one fails, and then disables itself again once it finds the process is running again. This is an overly complicated solution, but it is a start.

                  Now, If IPMONITOR did the array build and then the unique OID SNMP GETs behind the scenes, and we just  specified the process and parameters, this would be my low overhead, super efficient process monitoring. This could be done by calling the running process OID walk function when no process was found and call the array sub_function to get the paramaters and then, with to running SNMP GET with the PID once the process was found.

                  Worst case, if we were at least able to set the OID in the SNMP user experience (internally or externally), the rest could probably be worked out. 

                  Paul.