Why would I see huge CPU spikes in my monthly reports?

Hey Jeremiah,

I'm getting the same behavior on one of my Cisco ASA 5520 firewalls (but not the other two). 'Hadn't noticed it before, because I don't run many reports out of Orion, but one shows the same thing:

Average: 22%

Peak CPU Load: 32767%

In my case I see that the spike happened at 4pm on 4/16/2012. I've checked to see if Orion has any events around that time, but no dice (and no events, at all). And no changes to my Orion server (or the firewall) during that time either.

--Chris

Update: To be accurate, each of the two CPUs is showing 16383%:

Hmm, good to know! I have only seen these results on 5510s and 5520s.  I wonder if there's some weird bug in the IOS with the way they handle SNMP requests when they get swamped down. 

Interesting... Sounds like it might be a case for SW support (though reproducing it on demand is gonna be hard).

What version of ASA are you running on your 10s and 20s? On the one of mine that spiked, I'm running 8.4(2). My others are running 8.3(2), but didn't have a spike last month.

We are running 8.4(2) as well!  hmm maybe something with that exact version then! I'll reach out to SolarWinds and let you know what I find out.

Would you please post your ticket #'s and keep us updated with your progress?

Thanks!

DH

Just opened #339969 for this issue.  I will keep everyone posted with what I find out.

Thanks!

Here's the response I got back.  Since I'm studying for the SCP, this should give me some good experience messing around with UnDPs!

Thank you for contacting SolarWinds Support.

Basically NPM will report the data we gather from your ASA based on the ff OID

Will poll for the following OID for CPU and Memory Utilization

OID = cpmCPUTotal5MinRev   1.3.6.1.4.1.9.9.109.1.1.1.1.8
OID = cpmCPUTotal1minRev   1.3.6.1.4.1.9.9.109.1.1.1.1.7

If that is not accurate then I will advice to use below OID with UnDP application.

1.3.6.1.4.1.9.9.109.1.1.1.1.7.9 - 1 minute value
1.3.6.1.4.1.9.9.109.1.1.1.1.8.9  - 5 minutes values

The undp or custom poller will help us validate if the npm console matches the value we poll when we used undp.

You can see the tutorial on how to use the Universal Poller on solarwinds.com.

Create a Universal Device Poller (UnDP) in the Orion Platform - SolarWinds Worldwide, LLC. Help and Support

Hey Jeremiah,

If you still have the ticket open, can you ask them what variable type is used to store the CPU value? I'm just wondering if we're seeing these incorrect values because of the ASAs returning null data to a request (in which case Orion uses the max value of the "short int" variable type, 32767). Granted, that seems to be the combined total of the per-CPU values of 16383.333.... (I infer the ".33333" from the fact that adding them together gives us a rounded-up value of 32767 (most likely, 32766.666666...).

Using the CPU graph on your ASA(s) running 8.4(2), how often do you see it spike to 32767% in the last 30 days? In my case, it only happened once in 30 days, so we'll need to go historical on the UnDP data to compare.

<break> Okay, so while I was typing, I tried to setup the UnDP poller using those OIDs that support gave you. Testing them against either of my ASA versions, UnDP said they weren't supported. Let me know what you get from yours.

Thanks,

Chris

I'm just getting a chance again to dig into this, so I haven't come up with anything too useful yet.  In the last 30 days, it only happened once, and since the start of the year, it's happened every month but February.  I'll let you know what else I find out!

Just tried creating the UnDPs myself and they come back either a) not supported or b) no value was returned.  Sent a couple screenshots to the rep. and asked what variables are used.  I'll let you know the response.