21 Replies Latest reply on Jul 20, 2012 9:57 AM by cmgurley

    Why would I see huge CPU spikes in my monthly reports?

    Jeremiah Holland
    Jeremiah Holland May 2, 2012 12:47 PM

      I have an issue where every month I run a "Network CPU load" report and it usually says a few nodes (some are unique to a month, others come and go with the months, noticed it's mostly firewalls) will spike at 32767% CPU utilization.  Can anyone explain why that may be? If this is posted in the wrong area where would I post it?  Thanks!

        • Re: Why would I see huge CPU spikes in my monthly reports?
          cmgurley
          cmgurley May 2, 2012 1:33 PM (in response to Jeremiah Holland)

          Hey Jeremiah,

           

          I'm getting the same behavior on one of my Cisco ASA 5520 firewalls (but not the other two). 'Hadn't noticed it before, because I don't run many reports out of Orion, but one shows the same thing:

           

          Average: 22%

          Peak CPU Load: 32767%

           

          In my case I see that the spike happened at 4pm on 4/16/2012. I've checked to see if Orion has any events around that time, but no dice (and no events, at all). And no changes to my Orion server (or the firewall) during that time either.

           

          --Chris

           

          Update: To be accurate, each of the two CPUs is showing 16383%:

           

          16-Apr-201204:00 PM16383 %16383 %
            • Re: Why would I see huge CPU spikes in my monthly reports?
              Jeremiah Holland
              Jeremiah Holland May 2, 2012 3:56 PM (in response to cmgurley)

              Hmm, good to know! I have only seen these results on 5510s and 5520s.  I wonder if there's some weird bug in the IOS with the way they handle SNMP requests when they get swamped down. 

                • Re: Why would I see huge CPU spikes in my monthly reports?
                  cmgurley
                  cmgurley May 3, 2012 9:20 AM (in response to Jeremiah Holland)

                  Interesting... Sounds like it might be a case for SW support (though reproducing it on demand is gonna be hard).

                   

                  What version of ASA are you running on your 10s and 20s? On the one of mine that spiked, I'm running 8.4(2). My others are running 8.3(2), but didn't have a spike last month.

                    • Re: Why would I see huge CPU spikes in my monthly reports?
                      Jeremiah Holland
                      Jeremiah Holland May 3, 2012 3:04 PM (in response to cmgurley)

                      We are running 8.4(2) as well!  hmm maybe something with that exact version then! I'll reach out to SolarWinds and let you know what I find out.

                        • Re: Why would I see huge CPU spikes in my monthly reports?
                          DanielleH
                          DanielleH Administrator May 3, 2012 3:16 PM (in response to Jeremiah Holland)

                          Would you please post your ticket #'s and keep us updated with your progress?

                           

                          Thanks!

                          DH

                            • Re: Why would I see huge CPU spikes in my monthly reports?
                              Jeremiah Holland
                              Jeremiah Holland May 3, 2012 4:06 PM (in response to DanielleH)

                              Just opened #339969 for this issue.  I will keep everyone posted with what I find out.

                              Thanks!

                                • Re: Why would I see huge CPU spikes in my monthly reports?
                                  Jeremiah Holland
                                  Jeremiah Holland May 4, 2012 12:46 PM (in response to Jeremiah Holland)

                                  Here's the response I got back.  Since I'm studying for the SCP, this should give me some good experience messing around with UnDPs!

                                   

                                  Thank you for contacting SolarWinds Support.

                                  Basically NPM will report the data we gather from your ASA based on the ff OID

                                  Will poll for the following OID for CPU and Memory Utilization

                                  OID = cpmCPUTotal5MinRev   1.3.6.1.4.1.9.9.109.1.1.1.1.8
                                  OID = cpmCPUTotal1minRev   1.3.6.1.4.1.9.9.109.1.1.1.1.7

                                  If that is not accurate then I will advice to use below OID with UnDP application.

                                  1.3.6.1.4.1.9.9.109.1.1.1.1.7.9 - 1 minute value
                                  1.3.6.1.4.1.9.9.109.1.1.1.1.8.9  - 5 minutes values

                                  The undp or custom poller will help us validate if the npm console matches the value we poll when we used undp.

                                  You can see the tutorial on how to use the Universal Poller on solarwinds.com.

                                  http://www.solarwinds.com/resources/democenter/Orion/UniversalDevicePollerIntro/index.aspx


                                    • Re: Why would I see huge CPU spikes in my monthly reports?
                                      cmgurley
                                      cmgurley May 4, 2012 3:07 PM (in response to Jeremiah Holland)

                                      Hey Jeremiah,

                                       

                                      If you still have the ticket open, can you ask them what variable type is used to store the CPU value? I'm just wondering if we're seeing these incorrect values because of the ASAs returning null data to a request (in which case Orion uses the max value of the "short int" variable type, 32767). Granted, that seems to be the combined total of the per-CPU values of 16383.333.... (I infer the ".33333" from the fact that adding them together gives us a rounded-up value of 32767 (most likely, 32766.666666...).

                                       

                                      Using the CPU graph on your ASA(s) running 8.4(2), how often do you see it spike to 32767% in the last 30 days? In my case, it only happened once in 30 days, so we'll need to go historical on the UnDP data to compare.

                                       

                                      <break> Okay, so while I was typing, I tried to setup the UnDP poller using those OIDs that support gave you. Testing them against either of my ASA versions, UnDP said they weren't supported. Let me know what you get from yours.

                                       

                                      Thanks,

                                      Chris

                                        • Re: Why would I see huge CPU spikes in my monthly reports?
                                          Jeremiah Holland
                                          Jeremiah Holland May 8, 2012 1:04 PM (in response to cmgurley)

                                          I'm just getting a chance again to dig into this, so I haven't come up with anything too useful yet.  In the last 30 days, it only happened once, and since the start of the year, it's happened every month but February.  I'll let you know what else I find out!

                                          • Re: Why would I see huge CPU spikes in my monthly reports?
                                            Jeremiah Holland
                                            Jeremiah Holland May 8, 2012 1:42 PM (in response to cmgurley)

                                            Just tried creating the UnDPs myself and they come back either a) not supported or b) no value was returned.  Sent a couple screenshots to the rep. and asked what variables are used.  I'll let you know the response.

                                              • Re: Why would I see huge CPU spikes in my monthly reports?
                                                cmgurley
                                                cmgurley May 8, 2012 1:45 PM (in response to Jeremiah Holland)

                                                Thanks for the updates. Yeah, same here.

                                                  • Re: Why would I see huge CPU spikes in my monthly reports?
                                                    Jeremiah Holland
                                                    Jeremiah Holland Jun 8, 2012 2:48 PM (in response to cmgurley)

                                                    Sorry for the month long wait haha!

                                                     

                                                    So the engineer suggested I do a MIB walk on this, and found these related MIBs in use

                                                     

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.2.1      cpmCPUTotalPhysicalIndex.1 1

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.2.2 cpmCPUTotalPhysicalIndex.2 1

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 cpmCPUTotal5sec.1        6

                                                    CISCO-PROCESS-MIB      1.3.6.1.4.1.9.9.109.1.1.1.1.3.2 cpmCPUTotal5sec.2        6

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 cpmCPUTotal1min.1       4

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.4.2 cpmCPUTotal1min.2       4

                                                    CISCO-PROCESS-MIB      1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 cpmCPUTotal5min.1       4

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.5.2 cpmCPUTotal5min.2       4

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.9.1 cpmCPUMonInterval.1  5

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.9.2 cpmCPUMonInterval.2  5

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.10.1 cpmCPUTotalMonIntervalValue.1 6

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.10.2   cpmCPUTotalMonIntervalValue.2 6

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.11.1 cpmCPUInterruptMonIntervalValue.1   0

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.11.2 cpmCPUInterruptMonIntervalValue.2   0

                                                    CISCO-PROCESS-MIB      1.3.6.1.4.1.9.9.109.1.2.1.1.1.1.1 cpmProcessPID.1.1         1

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.1.1.5.1.1 cpmProcessTimeCreated.1.1      0

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.3.1.5.1.1 cpmProcExtUtil5SecRev.1.1         0

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.4.1.2.1.1 cpmCPURisingThresholdValue.1.1 70

                                                    CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.4.1.3.1.1 cpmCPURisingThresholdPeriod.1.1 1

                                                     

                                                    But the ones the tech suggested we use (

                                                    1.3.6.1.4.1.9.9.109.1.1.1.1.7.9 - 1 minute value

                                                    1.3.6.1.4.1.9.9.109.1.1.1.1.8.9  - 5 minutes values

                                                    1.3.6.1.4.1.9.9.109.1.1.1.1.8 - OID = cpmCPUTotal5MinRev 

                                                    1.3.6.1.4.1.9.9.109.1.1.1.1.7 - OID = cpmCPUTotal1minRev   ) were not listed in the MIB walk.  SW said we should reach out to Cisco for the appropriate MIBs, but not sure who at Cisco to contact.


                                                    I asked them if they know the best department, since I'm not sure if TAC would know or not.  Waiting to hear back.  Any luck, cmgurley?

                                                      • Re: Why would I see huge CPU spikes in my monthly reports?
                                                        cmgurley
                                                        cmgurley Jun 11, 2012 9:53 AM (in response to Jeremiah Holland)

                                                        Nothing on my end, unfortunately. In the last 30 days, I had another spike (just one) that varies some from the previous. At noon on May 12th, the CPU check showed 19114% on both CPUs. That doubles to 38228%, which blows my prior theory of a variable default/max.

                                                         

                                                        If you have SMARTnet on your ASA(s) and have time, give TAC a call. They'll help and get the info, even if it seems irrelevant to them. Monitoring is a legit reason.

                                                         

                                                        Thanks,

                                                        Chris

                                                          • Re: Why would I see huge CPU spikes in my monthly reports?
                                                            Jeremiah Holland
                                                            Jeremiah Holland Jun 26, 2012 11:51 AM (in response to cmgurley)

                                                            I reached out to TAC and they sent me

                                                            ftp://ftp.cisco.com/pub/mibs/supportlists/asa/asa-supportlist.html

                                                             

                                                            I couldn't make too much of it, but when I sent back

                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.2.1 cpmCPUTotalPhysicalIndex.1 1
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.2.2 cpmCPUTotalPhysicalIndex.2 1
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 cpmCPUTotal5sec.1 6
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.3.2 cpmCPUTotal5sec.2 6
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 cpmCPUTotal1min.1 4
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.4.2 cpmCPUTotal1min.2 4
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 cpmCPUTotal5min.1 4
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.5.2 cpmCPUTotal5min.2 4
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.9.1 cpmCPUMonInterval.1 5
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.9.2 cpmCPUMonInterval.2 5
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.10.1 cpmCPUTotalMonIntervalValue.1 6
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.10.2 cpmCPUTotalMonIntervalValue.2 6
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.11.1 cpmCPUInterruptMonIntervalValue.1 0
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.11.2 cpmCPUInterruptMonIntervalValue.2 0
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.1.1.1.1.1 cpmProcessPID.1.1 1
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.1.1.5.1.1 cpmProcessTimeCreated.1.1 0
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.3.1.5.1.1 cpmProcExtUtil5SecRev.1.1 0
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.4.1.2.1.1 cpmCPURisingThresholdValue.1.1 70
                                                            CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.4.1.3.1.1 cpmCPURisingThresholdPeriod.1.1 1


                                                            But none of them were supported on my asa.... whooo... I asked if they could escalate this issue, so we'll see where this goes...

                                    • Re: Why would I see huge CPU spikes in my monthly reports?
                                      LloydK
                                      LloydK Jun 26, 2012 6:55 PM (in response to Jeremiah Holland)

                                      We have a number of Cisco ASA 55xx as well running under code rev 8.4(3) and we see the high CPU spikes in both Orion and also in the ASA's log.

                                       

                                      In opening a case with Cisco on the issue they are basicly saying it is an issue with the ASA as evidenced by the entry in the ASA log and that we should do a code upgrade to rev 8.4(4)1.

                                      • Re: Why would I see huge CPU spikes in my monthly reports?
                                        jase4772
                                        jase4772 Jul 20, 2012 6:09 AM (in response to Jeremiah Holland)

                                        Hi,

                                         

                                        I've got the same issue but on 3 Windows servers. All three show a peak of 32767 % with an average of 32767%, 8230% and 2570%. I've just moved to 10.3 and when I ran these reports last month didn't get this.

                                         

                                        Thanks

                                        Jase