Configuration Download Performance

I have a daily job configured to download Cisco startup and running configurations. The job has a total of 109 devices spread across two pollers w/ an 80 / 29 split.

If the Simultaneous Downloads/Uploads setting is set to 10, or 5, the job produces timeouts like the one shown below. If I set Simultaneous Downloads/Uploads to 2, all backups within the jobs complete successfully.

ERROR: Cannot download Startup config : Unable to Log into Router : Timeout. Device IP:

I researched the timeout issue a bit and performed the following:

Deleted and recreated the job (no change)
Enabled session tracing (output below, seems like it's not getting banner response. Where does the 3 second response timeout come from?)

[5/10/2016 1:15:39 PM] --> hostname#

[5/10/2016 1:15:39 PM] ProcessLogin State: 0

[5/10/2016 1:15:46 PM] TimerTick: mstrData=<hostname#> State=3 - Connected to server - idle

[5/10/2016 1:15:46 PM] Pending Disconnect = False

[5/10/2016 1:15:46 PM] Pre-Commands: Waiting more than 3 seconds for response start sending pre-command if any...

[5/10/2016 1:15:46 PM] Sending to get a banner!

[5/10/2016 1:15:47 PM] <--

[5/10/2016 1:15:47 PM] -->

[5/10/2016 1:15:47 PM] --> hostname#

[5/10/2016 1:15:47 PM] ProcessLogin State: 0

[5/10/2016 1:16:10 PM] TimerTick: mstrData=<hostname#> State=3 - Connected to server - idle

[5/10/2016 1:16:10 PM] Pending Disconnect = False

[5/10/2016 1:16:10 PM] Pre-Commands: Waiting more than 3 seconds for response start sending pre-command if any...

[5/10/2016 1:16:10 PM] TimerTick: Send to CRLF get prompt again

[5/10/2016 1:16:10 PM] <--

[5/10/2016 1:16:10 PM] TimerTick: Login Timeout

[5/10/2016 1:16:10 PM] Disconnected - From: 1.1.1.1

NCM version is 7.4.1. I'm about to install hotfix 2 but it's not promising given the release notes.

Hotfix 2 addresses

* Download config job may crash if config does not contain standard characters.

The failures I'm experiencing occur on random devices and config types, and don't happen at all if I reduce simultaneous job count to 2. I would expect the failures to be consistent if there was an issue with standard characters.

Find more posts tagged with

ncm

backup job

Accepted answers

All comments

christopher.a.thornton

You could be getting slow response times from the devices as NCM is trying to get logged into them and download the config. Check out this thread as it may be able to help you.

NCM 7.4 issue with downloading configs - router timeout

rschroeder

I had similar issues until I tweaked a few things in NCM:

I reduced the simultaneous downloads from the default to six. That seemed to work well in my environment, given some of the distances between my NCM solutions and the switches & routers.
I changed the type of download for several devices after conferring with Cisco TAC and SW Support. Some devices worked well with SSH, others had to use TFTP or SCP to download the configurations. The larger the configuration, the less likely it would download successfully on a nightly basis--until I changed the mode. And example is a 10-blade Cisco 4510 chassis switch with 30 lines of commands per interface. Multiply 384 physical line card ports, plus the routing & supervisor interfaces and SVI's, etc., and the configuration file was timing out before it completed downloading via one protocol. Moving to a different protocol made the downloads complete in a tiny fraction of the time, and jobs began to have a great track record of success in backups.

You can search Thwack for similar threads; you'll come up with the various solutions that members have found worked well in their own environments.

Swift packets--and downloads!

Rick Schroeder

coen

Yep, adjusting the connection and prompt timeouts under NCM Settings > Protocol Settings worked. I switched the simultaneous downloads/uploads setting back to 10. Connection timeout is 180 seconds, prompt timeout is 60 seconds.

I don't think this is the true fix for my problem though; it seems more like a workaround as the issue isn't due to slow response times from my devices. We do have large configurations but they download fine with default protocol settings and simultaneous downloads/uploads set to 2.

In the scenario rschroeder gave below, he has large configurations that will not download when NCM is configured with the default protocol settings. So whether he has 6 simultaneous downloads or 1, with default protocol settings he can expect timeouts. That's not the behavior I'm observing. He also mentioned a need to adjust which protocols SolarWinds was using to download configs from some of his devices. The failures I'm receiving are occurring on random nodes that, for example, work fine on try 1 but fail on try 2. So that's not it either.

Digging in a bit more on my end, I reconfigured the backup job to download startup configs only. I increased simultaneous downloads/uploads to 30, kept connection timeout at 180 and prompt timeout at 60, then deleted all existing session trace files and ran the job.

The job produced 5 errors (timeouts). All 5 of those errors produced session trace files between 4KB - 6KB. Each of those files logged a 'TimerTick: Login Timeout' entry after 75 seconds in the same place. The script says it's "sending to get a banner", waits and eventually times out.

The higher the simultaneous downloads/uploads setting is, the more errors I receive.

That wait between 5:34:57 PM and 5:36:04 PM does not exist within session trace files for successful downloads.

[5/10/2016 5:34:51 PM] -----------------NCM 7.4.1-------------------

...

[5/10/2016 5:34:56 PM] ProcessLogin State: 0

[5/10/2016 5:34:56 PM] --->hostname#

[5/10/2016 5:34:56 PM] ProcessLogin State: 0

[5/10/2016 5:34:57 PM] TimerTick: mstrData=<hostname#> State=3 - Connected to server - idle

[5/10/2016 5:34:57 PM] Pending Disconnect = False

[5/10/2016 5:34:57 PM] Sending to get a banner!

[5/10/2016 5:34:57 PM] <--

[5/10/2016 5:34:57 PM] -->

[5/10/2016 5:34:57 PM] --> hostname#

[5/10/2016 5:34:57 PM] ProcessLogin State: 0

<wait>

[5/10/2016 5:36:04 PM] TimerTick: mstrData=<hostname#> State=3 - Connected to server - idle

[5/10/2016 5:36:04 PM] Pending Disconnect = False

[5/10/2016 5:36:04 PM] Pre-Commands: Waiting more than 3 seconds for response start sending pre-command if any...

[5/10/2016 5:36:04 PM] TimerTick: Send to CRLF get prompt again

[5/10/2016 5:36:04 PM] <--

[5/10/2016 5:36:04 PM] TimerTick: Login Timeout

[5/10/2016 5:36:04 PM] Disconnected - From: 1.1.1.1

With simultaneous downloads set to 30, if I increase prompt timeout to 160 seconds, even though the connection has already been established (i.e. prompt already received), I don't receive the timeouts. There's definitely a problem with this download/upload functionality. It's inefficient the way it is. I'm not setting prompt timeout to 160 seconds because my devices may take that long to give me a prompt (could you imagine). It's being set because NCM needs the time to play catch up.

With simultaneous downloads/uploads set to 10, session timeout at 180 seconds and prompt timeout at 60 seconds, 80 devices are backed up successfully in 7 minutes 52 seconds.

With simultaneous downloads/uploads set to 30, session timeout at 180 seconds and prompt timeout at 60 seconds, 72 devices are backed up successfully (8 failures) in 3 minutes 15 seconds.

With simultaneous downloads/uploads set to 30, session timeout at 180 seconds and prompt timeout at 160 seconds, 80 devices are backed up successfully in 4 minutes 31 seconds.

I read a few articles on this topic and it seems like most people are running something less than 10 simultaneous jobs due to these types of issues. I'm lucky to have a small device count. I wouldn't be surprised if these backups run for hours at some companies.

SolarWinds is an amazing tool though. Love it!