Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials

Issue with multiple firmware upgrades on Cisco 9300

I've been upgrading firmware on Cisco C9300 switches with scheduling the jobs. The issue I've been seeing is a few of the upgrades will go through and perform correctly. However, I'm seeing many other upgrades that are failing to work with this error message:

"ERROR: Unable to get image filename "cat9k_iosxe.17.12.06.SPA.bin" from firmware upgrade storage"

tftp server is running and all switches are allowed to access.

I've tried different upgrade schedules with the same time and tried to stagger the upgrades over a different increments, from 15 minutes to 30 minutes.

I went into the "Firmware upgrade settings" and "Maximum number of simultaneous upgrade operations to run" was set to 5 (default) and I changed the to 25, I have the space in my storage to support this.

I confirmed manually with the same commands that these switches can and will upgrade properly plus transfer the new firmware and it works on some of the scheduled upgrades at night but not all. For example, I ran 8 upgrades "jobs" and only 2 were successful, this was over a 3 hour period.

Questions

Can only 1 upgrade operation run at 1 time and it has to fully complete before starting another upgrade?

Will changing the number of simultaneous upgrades to a higher number actually allow me?

Can I only perform 1 switch at a time or can I do multiple, say 2-5 per upgrade job?

It's really frustrating and I thought NCM would be useful but it's making me lean towards another vendor to get these upgrades to perform correctly.

Find more posts tagged with

Cisco Catalyst 9300

upgrade failure

Accepted answers

All comments

verwin_u

hi @twzbowser

your issue needs couple of considerations.

in the job, is it all 9300 switches? do you have other device assigned in the job
do they have the same old version of ios? they might have differences in the prompts. which may lead to this kind of error and will fall into template issue.

for such issue, i highly recommend raising a support ticket to our ncm geeks and provide ncm logs and jobs log for proper assessment what went wrong.

https://customerportal.solarwinds.com/support/submit-a-ticket

cnorborg

We tend towards a bit more manual approach. We have simple scripts to load the software onto the box. Then we have another script that installs and reboots. The actual firmware upgrade process is a bit complex. The whole copying the software to the FTP directory and such tends to break down for us.

twzbowser

It was either 9300 or a few 9500s. I think the issue is with the tftp server, running on the main solar server was having issues.

The script works fine for 1 upgrade job, pushed it out to the lab and in production multiple times without an issue.

I did reach out to support, they suggested the {SubFolder} which was always recommended but now it's creating ghost folders on the tftp server that don't exists and the transfer fails since the switch cannot find the file. Trying the SCP route now which is more work but still it's not actually creating the folder.

Support is escalating the ticket but for now we are going to stick to small deployments and I'm looking down the external automation route for future upgrades. The NCM tool seemed great but a lot of hand holding is happening which defeats the purposes.

twzbowser

This is what I've thought about, preload the firmware on the switch first and then at night execute a script. The preloading can be cumbersome but at least I don't need to be up at 2am t

sja

NCM do not support back input.

It's not the right tool for that job.

cnorborg

Huh? Back input? Not sure what you're saying…

twzbowser

My work around was to pre-load the file and run "install add file bootflash:<filename> activate commit prompt-level none" as a scheduled job. This will activate the install package and commit it without needing a reply answer. The switch will automatically reboot.

Check that your switch is in install mode and verify the hash before letting the job run.

This works for smaller environments.