Cisco 6509 Logs full of: %SNMP-3-AUTHFAIL

[jagle]

Here's what my log file looks like on one of my core 6509s (imagine about a billion more though):

Jan 25 15:06:19: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 10.16.2.142
Jan 25 15:23:42: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 10.16.2.142
Jan 25 15:35:39: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 10.16.2.142
Jan 25 15:51:28: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 10.16.2.142
Jan 25 16:08:30: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 10.16.2.142
Jan 25 16:20:10: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 10.16.2.142
Jan 25 16:38:33: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 10.16.2.142

The node is perfectly managable via SNMP. BTW 10.16.2.142 is our polling engine.

1. At first glance it would seem to be an ACL issue (we have the community ACL'd) but that looks fine. 
   
2. Also the "Test" SNMP button in manage nodes works fine.  
   
3. I even went as far as to pull out wireshark, and I didn't see any packets at the time of the alert with the wrong community.
4. I disabled SNMP subnet scanning in IPAM, no change.

Dunno what else to look at...any thoughts?

APM 4.2.0 SP1, IPAM 2.0.1, IPSLAMGR 3.5.1, NCM 7.0, NPM 10.2.1, NTA 3.8.0

[wbrown]

I'm betting that you have your read-only string defined within NPM, but no read-write string.

In my environment I ended up pushing out "no logging snmp-authfail" to all my switches.  This solution was taken because we have other devices, managed by a separate group, that regularly perform network scans and test SNMP manageability.

[jagle]

I'm betting that you have your read-only string defined within NPM, but no read-write string.

That's absolutely true.  Although we've never defined one in NPM before, and never seen these logs. 

If true, wonder what it could be trying to do with r/w (if anything) on the switch?

I think I'll just not log it, in lieu of finding out.

[jswan]

I've seen this too. I didn't spend a bunch of time on it, but I did pull some packet captures to try to figure out what's going on. From the timestamps it looks like the offending packets are ones that have a different format in the community string. For example, if the SNMP community string is "foo", sometimes NPM sends requests where the community string is set to "foo@123" where, 123 is some three-digit number that changes across requests. The switch responds to these requests normally, so I don't know what's going on; I'm not up on the gory details of SNMP packet formatting. But it seems to only have started with the most recent NPM upgrade.

[savell]

The 3 digit number is a vlan on the device.

This type of polling is generated by the topology polling process (it will poll each vlan on the device).

Topology polling was enabled on supported devices by default in this release (although you can list resources on the device and unselect it if you don't want to collect the topology information for that device).

I have a issue in that the topology poller doesn't seem to handle unrouted valns (for example in DMZ environments). It causes disk space issues on my poller with huge numbers of topology polling errors in the Orion Core.Collector log files. I have a case open for this.

Dave. 

[wbrown]

Savell:  All good to know.

I'm still on NPM 10.1 and I've got just over a thousand (according to NCM) access-layer switches with a couple hundred unrouted vlans on each.  We're planning on moving to 10.2 in the next couple weeks.  Keep us updated on your disk space issue.