We have a job setup within NCM to download all the switch configs at 2am each morning.
Whilst this works great, we are seeing consistent errors in the switch log at the same time the configs download (between 2am and 3am).
The errors are:
May 16 02:12:49.166 BST: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
It seems as if NCM is not terminating the SSH connection after downloading the configs and the message above is once the timeout value is reached.
Anyone had this issue before?
No the errors do not appear if we do a manual download so seems to be related to the automatic scheduled ones.
It also doesnt happen every day, the odd few days complete without errors in the switch log.
Ever since we went from Telnet to SSH on NCM, some of our Cisco switches (especially those running v12.44 SE1 ) eventually run of memory and crash due to SSH sessions not disconnecting properly (memory chart looks like a memory leak, but crash file says ssh service is to blame).
Cisco says it's due to ssh clients not disconnecting properly and a fix is to upgrade the ios (sure enough, went to SE6 and problem went away).
So i'm not sure if Solarwinds NCM can do better job of disconnecting sessions to avoid having to upgrade all my switches ?.... or whether Cisco is to blame with certain bad versions of ios.
It's possible NCM could handle it better (I don't know all the details) but on the other hand, a reasonable device should not get injured by something as trivial as SSH client disconnecting improperly.
I would agree, but when you want to connect to devices many times over - like every hour, or even every half hour, having a message get logged gets very old very quickly.
Luckily, I don't see a memory leak in my devices, otherwise this could be catastrophic.
SolarWinds solutions are rooted in our deep connection to our user base in the THWACK® online community. More than 150,000 members are here to solve problems, share technology and best practices, and directly contribute to our product development process. Learn more today by joining now.