How to add memory utilization and CLI in top 10 process alert

rajasekar

How to add memory utilization and CLI in top 10 process alert

KMSigma.SWI

If we start with the alert action, it shows the following:

For the alert that displays the top 10 processes, it uses the SolarWinds.APM.RealTimeProcessPoller.exe executable on your SolarWinds server.

If you navigate to the server and get help (/?) on the program from the CLI, it provides the following:

SolarWinds.APM.RealTimeProcessPoller.exe -n= [-count=] [-sort=] [-timeout=] [-alertid=]

Command line arguments:
-n - defines ID of a Node (NodeID), which will be polled
-count - defines number of processes to show
-sort - defines criteria to use for selection of top processes. Possible values:
        CPU - processor time. This is default value if command line argument is not specified
        PhysicalMemory - process physical memory
        VirtualMemory - process virtual memory
        DiskIO - process disk I/O per second

-timeout - defines timeout for polling in seconds
-alertid - defines AlertID of associated triggered alert. If this argument is provided, then alert notes are updated with results from polling
-activeObject - defines ActiveObject property of associated triggered alert. If this argument is not provided, NodeID is used

Example:
    SolarWinds.APM.RealTimeProcessPoller.exe -n=123 -count=20 -sort=VirtualMemory -timeout=300

Example usage from 'Execute an External Program' alert action:
    SolarWinds.APM.RealTimeProcessPoller.exe  -n=${NodeID} -alertid=${N=Alerting;M=AlertID}

Example usage from 'Execute an External Program' alert action - for alert defined for 'Volume' object type:
    SolarWinds.APM.RealTimeProcessPoller.exe -n=${NodeID} -alertid=${N=Alerting;M=AlertID} -activeObject=${NetObjectID} -sort=VirtualMemory

If we go back to the execution of the action, you can see it uses the syntax (converted to multiple lines for reasability):

SolarWinds.APM.RealTimeProcessPoller.exe
   -n=${NodeID}
   -alert=${N=Alerting;M=AlertDefID}
   -alertId=${N=Alerting;M=AlertID}

Running it with just a node number (-n=1), returns the process list sorted by CPU. (the default)

System information:
        Processor count: 4
        Total Physical Memory: 13,931,436 kB
        Total Virtual Memory: 31,234,988 kB
        Total Process Count: 201

Name                                            Process ID      CPU
WmiPrvSE.exe                                    18060           21.35 %
lsass.exe                                       800             4.18 %
conhost.exe                                     15468           2.66 %
svchost.exe                                     2664            1.9 %
MsMpEng.exe                                     2256            1.29 %
SWJobEngineWorker2x64.exe                       10616           0.99 %
erl.exe                                         4420            0.68 %
SWJobEngineSvc2.exe                             4796            0.68 %
svchost.exe                                     468             0.61 %
SolarWinds.InformationService.ServiceV3.exe     5728            0.53 %

It can also be sorted other ways by adding in the -sort parameter.  Below, I run it with -sort=PhysicalMemory.

System information:
        Processor count: 4
        Total Physical Memory: 14,246,828 kB
        Total Virtual Memory: 31,550,380 kB
        Total Process Count: 204

Name                                                    Process ID      Physical Memory
w3wp.exe                                                3968            1,121,204 kB
SolarWinds.BusinessLayerHost.exe                        3316            974,988 kB
SolarWinds.InformationService.ServiceV3.exe             5728            686,744 kB
SolarWinds.Collector.Service.exe                        2960            620,740 kB
SolarWinds.ServiceHost.Process.exe                      2236            316,732 kB
MsMpEng.exe                                             2256            315,396 kB
SolarWinds.Administration.exe                           2880            303,608 kB
SolarWinds.BusinessLayerHost.exe                        10388           299,468 kB
SWJobEngineWorker2x64.exe                               10616           237,004 kB
NetFlowService.exe                                      14796           210,284 kB

It can also be run with DiskIO or VirtualMemory as a sort parameter.

The most important part in the Alert Action is to provide an Alert and an AlertID as parameters.  Why?

This is alert has escalations:

When the High CPU is detected, it runs the SolarWinds.APM.RealTimeProcessPoller.exe executable which stores the results in the Alert Notes for that triggered alert (because we included the Alert and AlertID as parameters).

The second escalation (which executes 2 minutes after the first) is what actually sends the email to the parties.  One of the variables in that message is the notes, updated in the previous escalation.

You can do the same thing for memory, virtual memory, and disk IO - just be sure to change the language and the alert triggers to match what's being expected.