12.2(33)SXI3 and IPSLA Manager?

NPM 10.1.1

IPSLA 2.5.1

We are rebuilding our Orion environment and started putting in IPSLA operations. Since we did this it *appears* that doing so booted our entire datacenter switching environment yesterday and today we just booted our main core switch after removing several IPSLA’s impacting 3000 people. I didn’t tread lightly because our current IOS didn’t appear to be on the naughty list of IPSLA’s that crash IOS.

I read this..

http://knowledgebase.solarwinds.com/kb/questions/331/Why+do+my+network+devices+reboot+when+I+try+to+create+IP+SLA+operations?

and

(our IOS ver is not mentioned)

We are running 12.2(33)SXI3 on a 6500VSS pair in both situations. Needless to say when you boot your datacenter one day and your core environment the very next day your outlook on life is not good.

Does anyone have any ideas here? Any way to have SW look at 12.2(33) as a potential victim of this problem? Do I need to open a support ticket?

Please advise.

-b

Find more posts tagged with

6500vss

reboot_cisco

ip_sla_manager

ipsla

i_want_my_mommy

cisco

6500_ipsa

ios_crash

pissed_off_boss

6500_crash

ipsla_crash

ios_ipsla

Accepted answers

All comments

JasonFerree

Cisco TAC may actually be your best bet. SW support won't be able to troubleshoot why your devices reset.

Bahlkris

Ok just want to make sure its an IOS problem and not IPSLA, and per that other link, glad its at least now "on the list". I'll report back what TAC says.

fcaron

Tks Bahlkris.

On our end, I'll ping dev and see what they might have seen in this area and report here.

Bahlkris

Ok so here is the official word from TAC on our 6500 issue. It appears the affect is broad and they have no intention to fix it. I dont believe there is a VSS version that isnt 12.2, so this would appear to affect anyone running VSS.

Removing ip sla probe (configured by SNMP) in CLI reloads Standby Sup
Symptom:

The standby supervisor reloads after removing an ip sla probe in CLI:

R7600(config)#no ip sla 1
R7600(config)#
06:53:31: Config Sync: Line-by-Line sync verifying failure on command:
no ip sla 1
due to parser return error

06:53:31: rf_reload_peer_stub: RP sending reload request to Standby. User:
Config-Sync, Reason: Configuration mismatch
R7600(config)#
06:53:31: %RF-SP-5-RF_RELOAD: Peer reload. Reason: Proxy request to reload peer
R7600(config)#
06:53:31: %OIR-SP-3-PWRCYCLE: Card in module 6, is being power-cycled (RF request)
R7600(config)#
06:53:32: %PFREDUN-SP-6-ACTIVE: Standby processor removed or reloaded, changing
to Simplex mode
R7600(config)#

Conditions:

This only occurs if the probe was configured via SNMP.

Workaround:

The standby supervisor doesn't reload if you remove the probe via SNMP.

Further Problem Description:

This issue is applicable to Catalyst 6500 platform running 12.2SX releases.
Yeah, I found a bug exactly like you’re talking about. The bug is CSCtd45679 and more information is available here:
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtd45679

Currently there is no bug fix and no ETA on a fix.. There is workaround where if you remove the probe with SNMP then the issue will not occur.

Unfortunately since there is no ETA on a bug fix, aside from using the workaround, there isn’t much else I can advise you to do. :-(

sallokatallo

We have the same SW products and three 6500 VSS pairs running the same code. I have been running IPSLA with no problem - added, modified and removed a few so far during the day no less!

I just want to try to get more detail.

If I understand correctly, the CSCtd45679 bug details say that the crash does not occur if the IPSLA configuration is removed via SNMP, and only occurs when the configuration is entered via SNMP, but removed via CLI. Was that your exact scenario?

I need to rebuild my environment as well!

I would love to speak with you if you don't mind talking to to me. Any clarification we can get can be added to this post, and it may help others as well. Send me a pm and we can get in touch

Salman

Bahlkris

So my understanding after having worked with Adrian and Francois is that all operations but the path operations(Path Echo and Path Jitter) are created, polled and deleted via SNMP. Only path echo and jitter are done via the CLI so my question to you would be what is your exact IOS version and what operations are you using?

I personally saw the problem happen 3 times, when we realized what was happening we put a freeze on it and didn’t touch anything. Then a power downtime got scheduled later and we used that window to remove all the IPSLA operations from all 4 of our 6500 VSS nodes at once. When we did that the crash did not happen. So either we were lucky, the crash happens when putting operations on or modifying them, or some other mitigating factor we don’t know about.

-b

ps. as someone who tends to be security paranoid id pull that phone number our of your reply.

Network_Guru

Rather than run IPSLA on our core or data center infra, we are purchasing 20 Cisco 1800 series routers as "shadow routers" in all our major call centers.
If these shadow routers crash - no harm, no foul.

This will give us visibility on many call center apps (mainly web based) as well as VoIP scores to the Cisco Voice Gateways in the Data Centers.
As a general rule, we will only run IPSLA responder on a production router (C3845s, AS5400s, etc).
There is too much at stake to enable IPSLA operations on a prodcution router.

bgfl-tech

Hi,

My organisation has just invested in the IP SLA Manager module as part of a larger upgrade of our NPM toolset to the latest release. We've also nearly finished a rolling upgrade of our core Cisco 6509 estate to IOS 12.2(33)SXI5 which after reading this thread puts the use of IP SLA operations across our core network into question.

The majority of 6509s are running as MPLS PEs and we're (were) looking to run IP SLA operations between these core nodes as well as between selected edge sites connected to those PEs.

Can I ask whether anyone has any knowledge of whether SXI5 also contains this bug and what workarounds (if any) people have implemented?

Also can anyone confirm exactly what causes the crash to happen as I'm still not totally sure from reading this thread? My current understanding is that if a probe is configured via SNMP using IP SLA Manager then removed manually from the CLI then you (may) get a crash but using IP SLA Manager exclusivily to add and remove probes is OK?

TIA

Matthew