Is there a way to automate firmware upload 1 device at a time?

We have hundreds of Cisco devices in our inventory and our network bandwidth is slow AF. My management do not allow us to upload more than 1-2 devices at a time so right now I'm manually creating 1 job per device that runs for the entire week.

Eg.

Job 1: Run script to upload firmware to Switch 1 at 00:00am

Job 2: Run script to upload firmware to Switch 2 at 01:00am

Job 3: Run script to upload firmware to Switch 3 at 02:00am

I know this looks insane and mindless but it's what I have to do, when I tried upload in batches of 4-5 devices at a time it usually fails. The upload speed is so slow that it takes avg 35-40min to upload a switch, and if it is congested it may take even longer which is why I schedule 1 switch per hour.

Once the upload is done, we run another script that installs and reload entire batch of devices. I have been in my current workplace for almost a year and that's how my team has always been doing things and now that it's my turn I can't help but feel there should be a better way to do it. Do you guys know a way how I can automate this upload process 1 device at a time instead of having to create hundreds of jobs manually?

Find more posts tagged with

scheduled_job

ncm

upgrade

script

firmware upgrades

Accepted answers

All comments

TerryHarding

When a firmware upgrade operation runs, the nodes targeted by that operation are upgraded one at a time, sequentially. In Firmware Upgrade Settings, you can limit simultaneous upgrades:

JasonChan94

Hi Terry,

Thanks for the suggestion, I've explored this earlier but doesn't know how it actually works and there are so few documentations regarding this feature.

I tried to do a test upgrade, removed basically every step of the upgrade except for the one that says "copy ${TransferProtocol} xxxxxx" I set it to use SCP, it ended up basically running the copy script to start the transfer on my Solarwinds SCP/SFTP server, but even though it runs one at a time that's only running the command, not taking into account the actual transfer time.

So when with my test upgrade of 2 switch, it did run the command one at a time, but without waiting for the transfer to finish, both switch was transferring the firmware simultaneously.

Maybe if I actually do the entire upgrade process, it will complete the whole process one at a time, but with just uploading the firmware I can't seem to get it to work the way I want it to. However the first option is not a solution for me because my management wants the firmware all uploaded and then the switches all reload at the same time during a specific maintenance window.

TerryHarding

I suspect it would complete one upgrade at a time. It seems what you need is a script to pre-stage the firmware files ready for the job to run for all devices during your maintenance window. I suspect you'll need to look outside of SolarWinds to achieve that.

vonzoom

What model switch are you updating, and what is the full command line in your template? You should be able to add multiple devices to a single job, which will perform the update (file transfer) serially. However, the drawback is that if a failure occurs, it will halt the remainder of the job.

sja

I will say take @TerryHarding advice.

Any how NCM is not optimal for that.

The lack of feedback/control can be a disaster..