-
Re: Why would I see huge CPU spikes in my monthly reports?
cmgurley May 2, 2012 1:33 PM (in response to Jeremiah Holland)Hey Jeremiah,
I'm getting the same behavior on one of my Cisco ASA 5520 firewalls (but not the other two). 'Hadn't noticed it before, because I don't run many reports out of Orion, but one shows the same thing:
Average: 22%
Peak CPU Load: 32767%
In my case I see that the spike happened at 4pm on 4/16/2012. I've checked to see if Orion has any events around that time, but no dice (and no events, at all). And no changes to my Orion server (or the firewall) during that time either.
--Chris
Update: To be accurate, each of the two CPUs is showing 16383%:
16-Apr-2012 04:00 PM 16383 % 16383 % -
Re: Why would I see huge CPU spikes in my monthly reports?
Jeremiah Holland May 2, 2012 3:56 PM (in response to cmgurley)Hmm, good to know! I have only seen these results on 5510s and 5520s. I wonder if there's some weird bug in the IOS with the way they handle SNMP requests when they get swamped down.
-
Re: Why would I see huge CPU spikes in my monthly reports?
cmgurley May 3, 2012 9:20 AM (in response to Jeremiah Holland)Interesting... Sounds like it might be a case for SW support (though reproducing it on demand is gonna be hard).
What version of ASA are you running on your 10s and 20s? On the one of mine that spiked, I'm running 8.4(2). My others are running 8.3(2), but didn't have a spike last month.
-
Re: Why would I see huge CPU spikes in my monthly reports?
Jeremiah Holland May 3, 2012 3:04 PM (in response to cmgurley)We are running 8.4(2) as well! hmm maybe something with that exact version then! I'll reach out to SolarWinds and let you know what I find out.
-
Re: Why would I see huge CPU spikes in my monthly reports?
DanielleHMay 3, 2012 3:16 PM (in response to Jeremiah Holland)
Would you please post your ticket #'s and keep us updated with your progress?
Thanks!
DH
-
Re: Why would I see huge CPU spikes in my monthly reports?
Jeremiah Holland May 3, 2012 4:06 PM (in response to DanielleH)Just opened #339969 for this issue. I will keep everyone posted with what I find out.
Thanks!
-
Re: Why would I see huge CPU spikes in my monthly reports?
Jeremiah Holland May 4, 2012 12:46 PM (in response to Jeremiah Holland)Here's the response I got back. Since I'm studying for the SCP, this should give me some good experience messing around with UnDPs!
Thank you for contacting SolarWinds Support.
Basically NPM will report the data we gather from your ASA based on the ff OID
Will poll for the following OID for CPU and Memory Utilization
OID = cpmCPUTotal5MinRev 1.3.6.1.4.1.9.9.109.1.1.1.1.8
OID = cpmCPUTotal1minRev 1.3.6.1.4.1.9.9.109.1.1.1.1.7
If that is not accurate then I will advice to use below OID with UnDP application.
1.3.6.1.4.1.9.9.109.1.1.1.1.7.9 - 1 minute value
1.3.6.1.4.1.9.9.109.1.1.1.1.8.9 - 5 minutes values
The undp or custom poller will help us validate if the npm console matches the value we poll when we used undp.
You can see the tutorial on how to use the Universal Poller on solarwinds.com.
http://www.solarwinds.com/resources/democenter/Orion/UniversalDevicePollerIntro/index.aspx
-
Re: Why would I see huge CPU spikes in my monthly reports?
cmgurley May 4, 2012 3:07 PM (in response to Jeremiah Holland)Hey Jeremiah,
If you still have the ticket open, can you ask them what variable type is used to store the CPU value? I'm just wondering if we're seeing these incorrect values because of the ASAs returning null data to a request (in which case Orion uses the max value of the "short int" variable type, 32767). Granted, that seems to be the combined total of the per-CPU values of 16383.333.... (I infer the ".33333" from the fact that adding them together gives us a rounded-up value of 32767 (most likely, 32766.666666...).
Using the CPU graph on your ASA(s) running 8.4(2), how often do you see it spike to 32767% in the last 30 days? In my case, it only happened once in 30 days, so we'll need to go historical on the UnDP data to compare.
<break> Okay, so while I was typing, I tried to setup the UnDP poller using those OIDs that support gave you. Testing them against either of my ASA versions, UnDP said they weren't supported. Let me know what you get from yours.
Thanks,
Chris
-
Re: Why would I see huge CPU spikes in my monthly reports?
Jeremiah Holland May 8, 2012 1:04 PM (in response to cmgurley)I'm just getting a chance again to dig into this, so I haven't come up with anything too useful yet. In the last 30 days, it only happened once, and since the start of the year, it's happened every month but February. I'll let you know what else I find out!
-
Re: Why would I see huge CPU spikes in my monthly reports?
Jeremiah Holland May 8, 2012 1:42 PM (in response to cmgurley)Just tried creating the UnDPs myself and they come back either a) not supported or b) no value was returned. Sent a couple screenshots to the rep. and asked what variables are used. I'll let you know the response.
-
Re: Why would I see huge CPU spikes in my monthly reports?
cmgurley May 8, 2012 1:45 PM (in response to Jeremiah Holland)Thanks for the updates. Yeah, same here.
-
Re: Why would I see huge CPU spikes in my monthly reports?
Jeremiah Holland Jun 8, 2012 2:48 PM (in response to cmgurley)Sorry for the month long wait haha!
So the engineer suggested I do a MIB walk on this, and found these related MIBs in use
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.2.1 cpmCPUTotalPhysicalIndex.1 1
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.2.2 cpmCPUTotalPhysicalIndex.2 1
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 cpmCPUTotal5sec.1 6
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.3.2 cpmCPUTotal5sec.2 6
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 cpmCPUTotal1min.1 4
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.4.2 cpmCPUTotal1min.2 4
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 cpmCPUTotal5min.1 4
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.5.2 cpmCPUTotal5min.2 4
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.9.1 cpmCPUMonInterval.1 5
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.9.2 cpmCPUMonInterval.2 5
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.10.1 cpmCPUTotalMonIntervalValue.1 6
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.10.2 cpmCPUTotalMonIntervalValue.2 6
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.11.1 cpmCPUInterruptMonIntervalValue.1 0
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.11.2 cpmCPUInterruptMonIntervalValue.2 0
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.1.1.1.1.1 cpmProcessPID.1.1 1
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.1.1.5.1.1 cpmProcessTimeCreated.1.1 0
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.3.1.5.1.1 cpmProcExtUtil5SecRev.1.1 0
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.4.1.2.1.1 cpmCPURisingThresholdValue.1.1 70
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.4.1.3.1.1 cpmCPURisingThresholdPeriod.1.1 1
But the ones the tech suggested we use (
1.3.6.1.4.1.9.9.109.1.1.1.1.7.9 - 1 minute value
1.3.6.1.4.1.9.9.109.1.1.1.1.8.9 - 5 minutes values
1.3.6.1.4.1.9.9.109.1.1.1.1.8 - OID = cpmCPUTotal5MinRev
1.3.6.1.4.1.9.9.109.1.1.1.1.7 - OID = cpmCPUTotal1minRev ) were not listed in the MIB walk. SW said we should reach out to Cisco for the appropriate MIBs, but not sure who at Cisco to contact.
I asked them if they know the best department, since I'm not sure if TAC would know or not. Waiting to hear back. Any luck, cmgurley?
-
Re: Why would I see huge CPU spikes in my monthly reports?
cmgurley Jun 11, 2012 9:53 AM (in response to Jeremiah Holland)Nothing on my end, unfortunately. In the last 30 days, I had another spike (just one) that varies some from the previous. At noon on May 12th, the CPU check showed 19114% on both CPUs. That doubles to 38228%, which blows my prior theory of a variable default/max.
If you have SMARTnet on your ASA(s) and have time, give TAC a call. They'll help and get the info, even if it seems irrelevant to them. Monitoring is a legit reason.
Thanks,
Chris
-
Re: Why would I see huge CPU spikes in my monthly reports?
Jeremiah Holland Jun 26, 2012 11:51 AM (in response to cmgurley)I reached out to TAC and they sent me
ftp://ftp.cisco.com/pub/mibs/supportlists/asa/asa-supportlist.html
I couldn't make too much of it, but when I sent back
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.2.1 cpmCPUTotalPhysicalIndex.1 1
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.2.2 cpmCPUTotalPhysicalIndex.2 1
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 cpmCPUTotal5sec.1 6
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.3.2 cpmCPUTotal5sec.2 6
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 cpmCPUTotal1min.1 4
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.4.2 cpmCPUTotal1min.2 4
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 cpmCPUTotal5min.1 4
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.5.2 cpmCPUTotal5min.2 4
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.9.1 cpmCPUMonInterval.1 5
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.9.2 cpmCPUMonInterval.2 5
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.10.1 cpmCPUTotalMonIntervalValue.1 6
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.10.2 cpmCPUTotalMonIntervalValue.2 6
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.11.1 cpmCPUInterruptMonIntervalValue.1 0
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.1.1.1.11.2 cpmCPUInterruptMonIntervalValue.2 0
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.1.1.1.1.1 cpmProcessPID.1.1 1
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.1.1.5.1.1 cpmProcessTimeCreated.1.1 0
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.3.1.5.1.1 cpmProcExtUtil5SecRev.1.1 0
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.4.1.2.1.1 cpmCPURisingThresholdValue.1.1 70
CISCO-PROCESS-MIB 1.3.6.1.4.1.9.9.109.1.2.4.1.3.1.1 cpmCPURisingThresholdPeriod.1.1 1But none of them were supported on my asa.... whooo... I asked if they could escalate this issue, so we'll see where this goes...
-
Re: Why would I see huge CPU spikes in my monthly reports?
Jeremiah Holland Jun 26, 2012 11:51 AM (in response to Jeremiah Holland)* When I sent the ftp link to solarwinds these mibs were the ones they suggested we try.
-
Re: Why would I see huge CPU spikes in my monthly reports?
cmgurley Jun 26, 2012 12:55 PM (in response to Jeremiah Holland)Hmm...maybe SW needs some 5520s running 8.4 in their lab . Or Cisco needs to run NPM... I can see each vendor favoring a different option
-
Re: Why would I see huge CPU spikes in my monthly reports?
Jeremiah Holland Jul 5, 2012 4:42 PM (in response to cmgurley)I just ran my reports, not changing anything, and to my surprise, i do not see a single utilization over 100%. Not sure on which side something got changed, but my last monthly report looks clean!! I'll still keep an eye on it, but it looks like (im guessing SW) someone did something to fix this bug haha!
-
Re: Why would I see huge CPU spikes in my monthly reports?
cmgurley Jul 6, 2012 9:17 AM (in response to Jeremiah Holland)Hmm..just checked my end, and the same (clean) results as you. Maybe 10.3 fixed it?
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Re: Why would I see huge CPU spikes in my monthly reports?
LloydK Jun 26, 2012 6:55 PM (in response to Jeremiah Holland)We have a number of Cisco ASA 55xx as well running under code rev 8.4(3) and we see the high CPU spikes in both Orion and also in the ASA's log.
In opening a case with Cisco on the issue they are basicly saying it is an issue with the ASA as evidenced by the entry in the ASA log and that we should do a code upgrade to rev 8.4(4)1.
-
Re: Why would I see huge CPU spikes in my monthly reports?
jase4772 Jul 20, 2012 6:09 AM (in response to Jeremiah Holland)Hi,
I've got the same issue but on 3 Windows servers. All three show a peak of 32767 % with an average of 32767%, 8230% and 2570%. I've just moved to 10.3 and when I ran these reports last month didn't get this.
Thanks
Jase
-
Re: Why would I see huge CPU spikes in my monthly reports?
cmgurley Jul 20, 2012 9:57 AM (in response to jase4772)What version of Windows? SNMP or WMI checks?
-