cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Additional Cisco "CPU & Memory" Poller for Cisco ASR Routers

Additional Cisco "CPU & Memory" Poller for Cisco ASR Routers

Request additional "CPU & Memory" poller to be available for use with Cisco ASR devices. This is due to being unable to duplicate the native "CPU & Memory" poller to update the specific memory OID. The reason for needing to do so is because we currently have various ASR routers falsely reporting high memory usage, when they are in fact operating within normal parameters.

Current list of memory OIDs polled by SolarWinds for Cisco can be found here for 1.3.6.1.4.1.9.9.109:

List_of_NPM_NTA_VNQM_MIBs_and_OIDs_used_for_Polling

CPU_and_memory_utilization_Object_IDs_OIDs_NPM_can_poll

This has been brought up with Cisco Support and their explanation is below.

================ ISSUE EXPLAINED ========================================

We have ASRs that show high memory usage at various time, as reported my our monitoring software SolarWinds. When I go check the command "show platform software status control-processor brief" it does have high usage. The devices will come and go from our dashboard.

I have followed this article, but not sure how to troubleshoot memory to determine if this is normal behavior verses an issue.

https://www.cisco.com/c/en/us/support/docs/routers/asr-1000-series-aggregation-services-routers/1167...

Is this normal, and if so, do you recommend adjusting the memory monitoring tool in SolarWinds to only flag if it hits 98% (97% is the highest I've seen)?

Command outputs below for the one device on the list.

See attached screenshot of what we see in SolarWinds.

=================================================

#show platform software status control-processor brief

Load Average

Slot  Status  1-Min  5-Min 15-Min

  RP0 Healthy   0.10   0.07   0.01

Memory (kB)

Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)

  RP0 Healthy  3969316  3849744 (97%)   119572 ( 3%)   2582596 (65%)

CPU Utilization

Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait

  RP0    0   2.90   7.20   0.00  89.90   0.00   0.00   0.00

         1   0.10   0.10   0.00  99.79   0.00   0.00   0.00

         2   0.00   0.00   0.00 100.00   0.00   0.00   0.00

         3   0.00   0.10   0.00  99.90   0.00   0.00   0.00

#monitor platform software process r0

top - 15:08:57 up 17 days, 22:13,  0 users,  load average: 0.04, 0.05, 0.00

Tasks: 305 total,   1 running, 304 sleeping,   0 stopped,   0 zombie

Cpu(s):  2.0%us,  1.6%sy,  0.0%ni, 96.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:   3969316k total,  3854584k used,   114732k free,   271512k buffers

Swap:        0k total,        0k used,        0k free,  2082440k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                    

23765 root      20   0 2804m 835m 351m S    2 21.5 190:56.19 linux_iosd-imag                                             

27281 root      20   0  141m 133m 6252 S    6  3.4   9:07.16 smand                                                       

31311 root      20   0 1826m 121m  64m S    5  3.1   1268:45 fman_fp_image                                               

30749 root      20   0 1361m 104m  41m S    0  2.7  16:53.77 cpp_cp_svr                                                  

Screenshot.png

================ RESPONSE FROM CISCO ========================================

Based on the screenshot and the commands attached, you are hitting a cosmetic issue described by CSCuc40262.

On examination of the router, we found that this “high” memory utilization is reflected in the output of 'show platform software status control-processor brief', which displays memory utilization from a system (Linux kernel) perspective:

 #show platform software status control-processor brief 
Load Average
Slot  Status  1-Min  5-Min 15-Min
  RP0 Healthy   0.10   0.07   0.01
Memory (kB)
Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
  RP0 Healthy  3969316  3849744 (97%)   119572 ( 3%)  2582596 (65%)

In this output, the "Committed" output is what we recommend focusing on, as this represents what memory processes have actually requested from the kernel. The "Used" value, on the other hand, appears high because this includes the Linux kernel cache: this "extra" memory is used by the kernel to store bits of frequently used data, but that memory can be freed at any time if needed. From the perspective of committed memory, this router is not low on memory and appears to be operating normally.

We frequently see cases inquiring about the misleadingly high value in the "Used" column. As a result, this is being adjusted in later code to provide a better representation of what memory is actually available for use. Additionally, two bugs have been filled to document the behavior, these are CSCuc40262 and CSCuv32343:

CSCuc40262    Linux kernel cache is leading to confusion over memory usage

https://tools.cisco.com/bugsearch/bug/CSCuc40262

CSCuv32343    show plat soft status contro [brief]: used memory's definitive accuracy

https://tools.cisco.com/bugsearch/bug/CSCuv32343

The first CCO release of IOS-XE with the full set of changes is the 3.17S release. This issue is mostly cosmetic and I would not personally consider this to be a pressing reason for an upgrade, you might want to apply the described workaround on the bug page instead.

If this issue is causing an alarm in network management software, the following OID should be used instead:

"cpmCPUMemoryCommitted" (1.3.6.1.4.1.9.9.109.1.1.1.1.27)
http://tools.cisco.com/Support/SNMP/do/BrowseOID.do?local=en&translate=Translate&objectInput=1.3.6.1...

================ END OF RESPONSE FROM CISCO ========================================

8 Comments

+1

We get bitten by this too, had to create an additional poller and remember to assign into all new ASR devices.

#bumpsquad  for some bump assistance on this

Level 8
Level 12

+ 1 this is a pain!

Level 12

Could you share your custom poller please mate? Cheers!

Sure, can't seem to attach the file so take the below and save it as a .poller file

<?xml version="1.0" encoding="utf-8"?>

<Poller xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.solarwinds.com/2008/DeviceStudio">

  <PollerID>00d13022-f6a6-496e-9570-efa85eed3446</PollerID>

  <TechnologyID>56112b7c-d1d7-4dc9-ae57-398116061379</TechnologyID>

  <PollingMethod>Snmp</PollingMethod>

  <Name>Cisco ISR Devices</Name>

  <Description></Description>

  <Author></Author>

  <Vendor></Vendor>

  <Tags></Tags>

  <Priority>500</Priority>

  <Enabled>true</Enabled>

  <Version>1</Version>

  <Configs xmlns:d2p1="http://schemas.datacontract.org/2004/07/SolarWinds.DeviceStudio.Common.Models">

    <contents xmlns:d3p1="http://schemas.microsoft.com/2003/10/Serialization/Arrays" xmlns="http://schemas.datacontract.org/2004/07/System.Collections.Specialized">

      <d3p1:KeyValueOfanyTypeanyType>

        <d3p1:Key xmlns:d5p1="http://www.w3.org/2001/XMLSchema" i:type="d5p1:string">inventory</d3p1:Key>

        <d3p1:Value xmlns:d5p1="http://www.w3.org/2001/XMLSchema" i:type="d5p1:string">&lt;Config xmlns:i="http://www.w3.org/2001/XMLSchema-instance" i:type="SnmpInventoryConfig" xmlns="http://schemas.solarwinds.com/2008/DeviceStudio"&gt;

  &lt;Operations xmlns:d2p1="http://schemas.microsoft.com/2003/10/Serialization/Arrays"&gt;

    &lt;d2p1:anyType i:type="OidExists"&gt;

      &lt;Oid&gt;

        &lt;OID&gt;1.3.6.1.4.1.9.9.221.1.1.1.1.18&lt;/OID&gt;

        &lt;Parser i:type="DefaultSnmpParser" /&gt;

      &lt;/Oid&gt;

      &lt;Method&gt;GetNext&lt;/Method&gt;

    &lt;/d2p1:anyType&gt;

    &lt;d2p1:anyType i:type="OidExists"&gt;

      &lt;Oid&gt;

        &lt;OID&gt;1.3.6.1.4.1.9.9.221.1.1.1.1.20&lt;/OID&gt;

        &lt;Parser i:type="DefaultSnmpParser" /&gt;

      &lt;/Oid&gt;

      &lt;Method&gt;GetNext&lt;/Method&gt;

    &lt;/d2p1:anyType&gt;

    &lt;d2p1:anyType i:type="OidExists"&gt;

      &lt;Oid&gt;

        &lt;OID&gt;1.3.6.1.4.1.9.9.109.1.1.1.1.8&lt;/OID&gt;

        &lt;Parser i:type="DefaultSnmpParser" /&gt;

      &lt;/Oid&gt;

      &lt;Method&gt;GetNext&lt;/Method&gt;

    &lt;/d2p1:anyType&gt;

  &lt;/Operations&gt;

&lt;/Config&gt;</d3p1:Value>

      </d3p1:KeyValueOfanyTypeanyType>

      <d3p1:KeyValueOfanyTypeanyType>

        <d3p1:Key xmlns:d5p1="http://www.w3.org/2001/XMLSchema" i:type="d5p1:string">polling</d3p1:Key>

        <d3p1:Value xmlns:d5p1="http://www.w3.org/2001/XMLSchema" i:type="d5p1:string">&lt;Config xmlns:i="http://www.w3.org/2001/XMLSchema-instance" i:type="PollingConfig" xmlns="http://schemas.solarwinds.com/2008/DeviceStudio"&gt;

  &lt;DataSourceCreateConfig i:type="SnmpCreateConfig"&gt;

    &lt;Operations xmlns:d3p1="http://schemas.microsoft.com/2003/10/Serialization/Arrays"&gt;

      &lt;d3p1:anyType i:type="SnmpGetTable"&gt;

        &lt;Table&gt;cempMemPoolTable&lt;/Table&gt;

        &lt;Columns&gt;

          &lt;Oid&gt;

            &lt;Name&gt;cempMemPoolHCUsed&lt;/Name&gt;

            &lt;OID&gt;1.3.6.1.4.1.9.9.221.1.1.1.1.18&lt;/OID&gt;

            &lt;Parser i:type="DefaultSnmpParser" /&gt;

          &lt;/Oid&gt;

          &lt;Oid&gt;

            &lt;Name&gt;cempMemPoolHCFree&lt;/Name&gt;

            &lt;OID&gt;1.3.6.1.4.1.9.9.221.1.1.1.1.20&lt;/OID&gt;

            &lt;Parser i:type="DefaultSnmpParser" /&gt;

          &lt;/Oid&gt;

        &lt;/Columns&gt;

      &lt;/d3p1:anyType&gt;

      &lt;d3p1:anyType i:type="SnmpGetTable"&gt;

        &lt;Table&gt;cpmCPUTotalTable&lt;/Table&gt;

        &lt;Columns&gt;

          &lt;Oid&gt;

            &lt;Name&gt;cpmCPUTotal5minRev&lt;/Name&gt;

            &lt;OID&gt;1.3.6.1.4.1.9.9.109.1.1.1.1.8&lt;/OID&gt;

            &lt;Parser i:type="DefaultSnmpParser" /&gt;

          &lt;/Oid&gt;

        &lt;/Columns&gt;

      &lt;/d3p1:anyType&gt;

    &lt;/Operations&gt;

  &lt;/DataSourceCreateConfig&gt;

  &lt;DataSourceTransformConfig i:type="TransformConfig"&gt;

    &lt;Operations xmlns:d3p1="http://schemas.microsoft.com/2003/10/Serialization/Arrays"&gt;

      &lt;d3p1:anyType i:type="Expression"&gt;

        &lt;Output&gt;

          &lt;Property&gt;UsedMemoryFormula1&lt;/Property&gt;

        &lt;/Output&gt;

        &lt;Expression&gt;Sum([cempMemPoolTable.cempMemPoolHCUsed])&lt;/Expression&gt;

      &lt;/d3p1:anyType&gt;

      &lt;d3p1:anyType i:type="Expression"&gt;

        &lt;Output&gt;

          &lt;Property&gt;FreeMemoryFormula1&lt;/Property&gt;

        &lt;/Output&gt;

        &lt;Expression&gt;Sum([cempMemPoolTable.cempMemPoolHCFree])&lt;/Expression&gt;

      &lt;/d3p1:anyType&gt;

      &lt;d3p1:anyType i:type="Expression"&gt;

        &lt;Output&gt;

          &lt;Property&gt;CPULoadFormula1&lt;/Property&gt;

        &lt;/Output&gt;

        &lt;Expression&gt;avg([cpmCPUTotalTable.cpmCPUTotal5minRev])&lt;/Expression&gt;

      &lt;/d3p1:anyType&gt;

    &lt;/Operations&gt;

  &lt;/DataSourceTransformConfig&gt;

  &lt;DataSourceOutputConfig i:type="OutputConfig"&gt;

    &lt;Properties&gt;

      &lt;OutputProperty&gt;

        &lt;Name&gt;Used Memory&lt;/Name&gt;

        &lt;Mapping&gt;UsedMemoryFormula1&lt;/Mapping&gt;

        &lt;Type i:type="Double" /&gt;

      &lt;/OutputProperty&gt;

      &lt;OutputProperty&gt;

        &lt;Name&gt;Free Memory&lt;/Name&gt;

        &lt;Mapping&gt;FreeMemoryFormula1&lt;/Mapping&gt;

        &lt;Type i:type="Double" /&gt;

      &lt;/OutputProperty&gt;

      &lt;OutputProperty&gt;

        &lt;Name&gt;CPU Load&lt;/Name&gt;

        &lt;Mapping&gt;CPULoadFormula1&lt;/Mapping&gt;

        &lt;Type i:type="Integer" /&gt;

      &lt;/OutputProperty&gt;

    &lt;/Properties&gt;

    &lt;Tables /&gt;

  &lt;/DataSourceOutputConfig&gt;

&lt;/Config&gt;</d3p1:Value>

      </d3p1:KeyValueOfanyTypeanyType>

    </contents>

  </Configs>

</Poller>

Level 12

Thanks! So where does this need to be saved and how do I replace the default CPU poller for ASR's with this one? How to implement? Thanks for taking the time @dsimpkins

Save it to your desktop and then from "manage pollers" import it.

Once there you can assign it to your nodes.

Community Manager
Community Manager
Status changed to: Open for Voting