*** Resolved Switch Stack IOS Upgrade with Change Control Template***

Version 1

    This document is Cisco centric.  I haven't had the opportunity to try this with other devices like Brocade, Extreme or HP

     

    The ‘Firmware Upgrade’ menu choice is not an option that we can use.  When we perform an IOS upgrade very rarely will we upgrade a standalone switch or even use the ‘copy’ command.  Unfortunately, this is how the ‘Firmware Upgrade’ menu choice in Orion works.  When we are upgrading our IOS switches they are always in stacks of 2 or more and we are usually  interested in upgrading just the .bin file and our goto command is the ‘archive download-sw’ command.  The 'archive download-sw' command uses the .tar file and the file respository for the ‘Firmware Upgrade’ menu choice will not recognize a .tar file.  Additionally when you use the ‘Copy’ command and the task is to upgrade a stack of switches you need to ‘Copy’ the .bin file to each switch in the stack individually, however,  when you use the ‘archive download-sw’ command the .bin file will be extracted to all switches in the stack as well as the boot variable updated for the entire stack with one command.

     

    The NCM Config Change Template script at the end is our work-around  for the limitations of the ‘Firware Upgrade’ Menu choice.  So far I have only tested this with Cisco IOS stacks that are the 2960S and the 2960X and roughly around 51 switch stacks have been upgraded...

     

    There is one other NCM limitation that you will need to keep in mind if you use a script like this to upgrade IOS devices in your environment.  That second limitation is NCM will disconnect from the switch if there is no response after a short period of time.  Anyone who has done an image upgrade on a switch stack before knows that this procedure can take some time. Lots of time, it’s 20 or 30 minutes, or possibly longer depending on the size of the stack.  When this happens, you will  receive a completed successfully message in the NCM  ‘Transfer Status Screen’ however the upgrade is far from over as you can see in the scripts result window once you open it up.

    A example of the script results window when NCM disconnects before end of upgrade is below.

    The work around is to max out the settings under the CLI.  (The CLI settings can be found in All Settings -> Product Specific Settings -> CLI).  I need to mention here that these settings ARE NOT recommended by SolarWinds and could potentially lead to other problems if left this way.  Once the upgrade is complete these settings need to be restored to their original values. I have pasted a screen shot of the CLI settings below:…

    Now when the script below is run NCM will not disconnect early and the results are displayed more accurately in the ‘Transfer Status’ Page.  A screen shot of the transfer status is pasted below when using the CLI settings above:…

     

    With the screen shots above its easy to see what is missed when NCM disconnects too early.  When NCM disconnects early the follow up commands are not executed.  This includes the 'write mem' command at the end of the script.  If jthe write mem command isn't executed then a reload of the switch could result in a failed upgrade of the stack.

     

    I used this procedure on few dozen switch stacks and all have completed successfully.  Remember I need to stress that the CLI settings I have shown above ARE NOT recommended by SolarWinds and need to be reset when the upgrades are done or are not being performed to avoid other problems.  For me this is not a big deal we just open the CLI settings make the adjustments and start the upgrade process.  When all nodes are complete we simply set the settings back again.   So far the largest group of nodes I have selected for an upgrade was 15 and the largest stack size in that group was 4. Total time to run through 2 groups of 15 was just about 7.5 hours.  If I upgrade one switch stack with 3 switches it usually takes 55 minutes.

     

     

    My choices for the IOS upgrade script...

     

     

    Script for IOS upgrade…..

     

    script UpgradeIOS (

                             NCM.Nodes @ContextNode,

                             string @IOS_FILENAME )

    {

     

     

    //lets generate the CLI command that will upgrade the switch from the info gathered.

     

     

    string @CommandLine = 'archive download-sw /imageonly /leave-old-sw tftp://${StorageAddress}/' + @IOS_FILENAME

    string @Enter = '${CRLF}'

     

     

    //Make sure there are no spaces in filename

    if (@IOS_FILENAME contains ' ')

    {}

     

    else

     

    // Make sure the switch doesn't already have the correct OS Version

    if ( @ContextNode.OSVersion StartsWith '15.2(2)E5' )

    {}

     

    else

     

    {

      foreach ( @node in @ContextNode )

     

      {

          if (@node.OSImage StartsWith 'C2960X-')

           {

            CLI

             {

              @CommandLine

             }

            CLI

             {

              @Enter

              @Enter

              Show boot

              @Enter

              @Enter

             }

            CLI

             {

              dir

              @Enter

              @Enter

             }

            CLI

             {

              wr me

              @Enter

             }

           }

      }

    }

    }