FEATURE REQUEST – Increase NCM Firmware Upgrade Post-Reboot Verification Timeout Beyond 120 Seconds

grzegorzj

Hello THWACK Community,

I would like to submit a feature request regarding the NCM Firmware Upgrade module and its post-reboot verification process.

SolarWinds Feature Request: CUST-129407

During the implementation and testing of automated firmware upgrades for Cisco network devices, we identified a limitation that significantly impacts unattended firmware lifecycle automation.

Currently, the maximum value for:

"Wait time between reboot and checking that node is UP"

is limited to 120 seconds.

For devices that require longer boot times, this creates a situation where:

• Firmware image transfer completes successfully.
• Boot variables are updated successfully.
• Configuration is saved successfully.
• The device reboots successfully.
• The new software version boots successfully.
• SSH services are not yet available within 120 seconds.
• NCM attempts post-reboot verification.
• The operation fails with a "Connection Refused" error and is reported as failed.

In our testing, Cisco Catalyst C1000 devices typically require approximately 160 seconds before SSH becomes available again.

The impact becomes even greater for enterprise platforms such as:

• Cisco Catalyst 9300
• Cisco Catalyst 9500
• Cisco Nexus
• Arista switches
• Other enterprise-class network devices

where reboot, initialization, package verification, stack initialization, and service startup can take several minutes before management access becomes fully available.

The main concern is that this limitation prevents reliable unattended firmware upgrade workflows.

Although SolarWinds currently recommends separating:

• firmware deployment,
• reboot,
• verification

into separate jobs, this approach removes the end-to-end automation that many customers expect from the NCM Firmware Upgrade feature.

We also confirmed with SolarWinds Support that running multiple firmware upgrade operations in parallel does not solve the underlying issue. Every operation eventually encounters the same 120-second verification limitation on devices with longer boot times.

From our perspective, a more scalable solution would be one of the following:

• Increase the maximum configurable timeout to at least 600-1000 seconds.
• Allow timeout configuration per node, device type, or upgrade template.
• Implement a retry-based verification mechanism that periodically checks SSH connectivity until a configurable timeout is reached.

Example:

"Attempt SSH verification every 10 seconds for up to 10 minutes."

This would allow the NCM Firmware Upgrade module to support modern network platforms more effectively and make unattended firmware lifecycle automation significantly more reliable.

If other customers have experienced similar behavior or would benefit from a configurable post-reboot verification timeout, please feel free to comment and support this feature request.

Thank you.

SolarWinds Observability Self-Hosted Advanced Enterprise Scale: 2025.2.1

Feature Request Reference: CUST-129407

cnorborg

I assume you've tried typing in a higher value rather than using the slider?

Yea, we aren't a fan of the actual upgrade process TBH. To limiting and difficult to use. We just write some scripts, one to preload the IOS on the devices. Another to install and reboot when we're ready. Then we look for down nodes on the dashboard.