Cisco Nexus fc interface showing warning state intermittently

We just deployed two pairs of Cisco Nexus 5548UP switches and have a couple 8Gbps fiber SFPs connecting to some HP 3PAR storage (doesn't support CNAs yet). Ever since we deployed, I see the first of the two fiber uplinks (fc1/31) pop up on my Orion NPM "Nodes with Problems" list intermittently. Just that interface, but on all of the switches at different times (sometimes concurrently; i.e. both of one pair this morning).

The state is "warning", but response time is normal (<1ms) and packet loss is 0%. When I drill down into the switches and interfaces, I see no reason for the warning state as no packets are being discarded, no errors recorded, no CPU or memory issues, and no interface saturation.

Does anyone know why Orion is setting this warning state? It doesn't log any events when this happens either from which to derive state change.

Thanks,

Chris

Find more posts tagged with

nexus

npm

cisco

fiber

Accepted answers

All comments

badger

I'm also having a similar issue, but this is with port-channels from a 5548P (not UP) running NX-OS 5.2(1)N1(1)

cmgurley

Same code version as I'm running. Hmm... 'Wish Orion had a better audit log and wrote to it with an explanation for warning states.

bill.h

I have a similar issue, only with Nexus 7ks...posted here: http://thwack.solarwinds.com/thread/52410

Seems to be some sort of issue with NX-OS and SNMP maybe....is everyone running v3 with auth and priv or are there different SNMP configurations out there?

badger

yup; SNMP v3 (auth, priv)

Orion Core 2012.1.0, NCM 7.0.2, NPM 10.3, IVIM 1.3.0, VNQM 4.0

cmgurley

Nah, I'm running v2 on mine.

orionshark

Did anyone solve this?

I am seeing some flapping interfaces on our Nexus 5548 & 5596UP nodes too.

It seems like these boxes are stressed by the SNMP monitoring somehow....

NPM10.4.1 and running SNMPv2C against the Nexus nodes.

orionshark

OK - I solved the problem by changing to SNMPv1 on the Nexus nodes.

Explanation: One of the changes in the new versions of NPM is that NPM use SNMP GETBULK requests when polling a node. Some devices do not like it and will fail to respond. To correct this, set the device to SNMP v1 and NPM will poll using SNMP GET requests instead. This solved my issue.

RichardLetts

One thing to look into here is the availability (or otherwise) of Jumbo frames in your environment. The SNMP getbulk request-pdu is normally a very small packet (less than 512 bytes), however the response may often be much larger, especially on the interfaces table where the interface descriptions are returned for up to 40 interfaces at once. If the sending device has jumbo frames configured then the getbulk reply-pdu may be larger than 1500 bytes, and be fragmented (if a L3 device) or discarded (by a L2-device) in the network.

Some switches (e.g. the HP5406) default to jumbo frames for the management interface if its interfaces are configured for jumbo frames, even if the router that it is connected to does not support jumbo frames. Juniper EX-series switches default the management interface to 1500 bytes even if the L2 interfaces are configured for jumbo frames.

As you say, switching to SNMPv1 disables get-bulk; I've yet to find a getbulk implementation that is actually broken on a modern switch/router.

You can test this out using the SNMPwalk utility and setting a high number of repeaters, and testing lower values until it works consistently.

FormerMember

This seems to tie in with this thread:

Has anyone here tried the SNMP v1 "solution"? I have the same issue as everyone else here, but haven't tried snmpv1.