Interface InErrorsThisHour

I need to resolve an issue that been reported on a node, either a router has an issue, I am gathering something unimportant or incorrectly, or I need to understand it better. Likely 2 of the 3. One interface of this router (which contents this offsite via GigaMAN service back to our primary datacenter). InErrorThisHour values range from 1 million to 4 million. I think there is clearly something wrong, but when the network engineer goes to check stats from the router (his counters haven't been reset in about 2 months) ther appear to be no errs. I feel like we aren't speaking the same language, any suggestions?

Pull these values using custom query for my network team:

SELECT

'' AS n,

'/Orion/images/StatusIcons/small-' + ToString(AdminStatusLED) AS [_IconFor_n],

FullName as [INTERFACE],

'/Orion/Interfaces/InterfaceDetails.aspx?NetObject=I%3a' + ToString(InterfaceID) AS [_LinkFor_INTERFACE],

MAC,

InErrorsThisHour as ['In Err HR'],

InErrorsToday as ['In Err Today'],

OutErrorsThisHour as ['Out Err HR'],

OutErrorsToday as ['Out Err Today']

FROM

Orion.NPM.Interfaces

Where

(OutErrorsToday + InErrorsToday) > 0

Order By (OutErrorsToday + InErrorsToday) Desc

This is the top entry all the time by a wide margin.

Find more posts tagged with

Accepted answers

brandon.blaze

It's a possibility that it could be a code issue on the router. Changing code revisions on a WAN device, especially if it's your only perimeter router can be a chore, what with possible downtime, verifying functionality, etc. Have you tried using a different MIB for polling the router? It may be using a generic Cisco MIB, or not be matched specific to the router and that could be sending up incorrect stats? Since you're not seeing it on other interface(s) it would lead me to believe that it's most likely an issue on the PE, rather than CE.

All comments

brandon.blaze

Is the node that's sending data to the router's interface that's reporting the errors something that's monitored by NPM? Does that device have any OutErrors when he runs a "show int count err" (assuming it's a Cisco device that's connected to the router)? Have all necessary cables been verified error free, or have they been replaced with fresh cables as a troubleshooting step? Is it possible that the router isn't classifying the errors as "InErrors" and perhaps they're Frame Check or CRC errors, but NPM is picking them up as a generic "InError" due to SNMP or something?

Suppose I should ask the router brand/vendor. I just assumed Cisco, but I could be way off.

jm_sysadmin

The next hop is Time Warner's device, and its what I assume is the issue, or the cables to it, but he hasn't tried thing like checking cables since his numbers for "show int count err" don't match. I asked for the output of the commands below to see if I can see why things are different, or if anything else doesn't align with what I think I know. The only other option I can guess besides the cables and next device is that we have a bug somewhere in the software, maybe in what populates SNMP.

show version

show module <mod#>

show running-config

show log

show interfaces fastethernet <mod/port> status

show interfaces fastethernet <mod/port> trunk

show interfaces fastethernet <mod/port> switchport

show mac-address-table dynamic interfaces fastethernet <mod/port>

show spanning-tree interfaces fastethernet <mod#/port>

show ip route

show ip arp

show ip [eigrp/ospf] neighbors

show cdp neighbors fastethernet <mod/port> detail

Repeat the following five commands three times to monitor counter increments:

show interfaces fastethernet <mod/port>

show interfaces fastethernet <mod/port> counters

show interfaces fastethernet <mod/port> counters errors

show interfaces fastethernet <mod/port> counters trunk

show interfaces fastethernet <mod/port> counters broadcast

diagnostic level complete (global configuration command)

show diagnostic module <mod#>

brandon.blaze

jm_sysadmin

This may have helped, I swapped to a different MIB. I am going to let it run for a day and I will let you know. Thanks!

jm_sysadmin

The MIBs didn't correct anything, we did find a QoS setting that needed to be corrected, but nothing made the errors go away. It does need updates, and we plan to do that, but it may take a bit to get it scheduled. I am going to mark the code issue on the router as correct, by eliminating the other options, its all we have left.

brandon.blaze

Well best of luck to you on the code upgrade, and hopefully that fixes the issue!

RichardLetts

what does Show Interface actually report?

On some routers they count 'L3 incomplete' as errors in the SNMP MIB, also if there is no default vlan and its discarding frames without the right set of tags

e.g. if this is connected to a MOE service and Time Warner are sending you data for a wrong VLAN, or you're seeing CDP, or clients sending non-IP packets

On our Juniper routers this shows up like this:

Input errors:

Errors: 5050812, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0,

L3 incompletes: 5050812, L2 channel errors: 0, L2 mismatch timeouts: 0,

FIFO errors: 0, Resource errors: 0

this is the graph for that interface (this is traffic from a wireless controller, so it's probably odd non-IP packets from a few of the hundreds of thousands of wireless clients.)