22 Replies Latest reply on Jul 6, 2016 8:32 AM by RichardLetts

    Orion Upgrades with FOE

    derik.pfeffer

      Is there any documentation to how "Upgrades" are handled with Orion products and FOE? I have 2 situations, first is using a single SQL Server and the other with either SQL using Cluster Option or the NeverFail solution.

      Best Regards,

      Derik

        • Re: Orion Upgrades with FOE
          bshopp

          Since Orion on upgrades typically needs access to the DB you are going to encounter downtime in a single DB setup/scenario.

          Here are some KB's that are worth checking out here and here

            • Re: Orion Upgrades with FOE
              beijota2

              What about when upgrading NPM?  (i.e. 10.1 to 10.2)

                • Re: Orion Upgrades with FOE
                  aLTeReGo

                  This KB article How to Upgrade SolarWinds Orion when Orion Failover Engine is Installed should guide you through the process.

                    • Re: Orion Upgrades with FOE
                      beijota2

                      That KB needs to be re-written, alterego.  Followed the instructions to the best of what I could interpret them to mean and ended up with a disaster.  2 weeks later we're still trying to recover.

                        • Re: Orion Upgrades with FOE
                          aLTeReGo

                          Where did the KB deviate from expected or what specific issues did you encounter while following them? We strive to make our documentation a clear, concise, and above all, accurate; but if you've found it to be incorrect or difficult to follow we'd love to get your detailed feedback so it can be improved. 

                        • Re: Orion Upgrades with FOE
                          mikegale

                          I know this is a really old thread but the instructions are a bit confusing.

                          Why can Orion upgrades not be handled like OS patches? For patches I install on Primary, fail over and install on Secondary. The confusing part in the instructions is the part where one has to make changes in the Config Server Wizard. The instructions say to change the current role to Secondary/passive but the "Machine" tab in the config server wizard already looks as follows:

                          Machine_tab.jpg

                          Does that mean no change is needed? If the instructions read change "Physical Hardware Identity" to Secondary and change "Active Server" to Secondary, no one would pause at this step. There's a couple more later on when one switches back over to the Primary server to run the install that could be changed.

                           

                          There's a section that reads "For the Secondary Server:" that contains a bunch of stuff about the primary server. The confusing part here is which server am I working on when it doesn't specifically mention it? Step 6 says to Restart FoE on the primary machine but step 7 just says to right-click and turn FoE back on. Does that mean I'm still working on the primary or does it mean the secondary because the step is under the section labeled "For the Secondary Server:"?

                           

                          If the section "For the Secondary Server:" could be reworked that'd be great. If one could perform upgrades the same way we install patches that'd be even better.

                           

                          Thanks

                          Michael

                            • Re: Orion Upgrades with FOE
                              RichardLetts

                              I find the instructions exact, but somewhat confusing. At step7 you are working on the secondary server, you are telling it to make the 'primary' (non-running) server as active.

                              then you start up neverfail on the secondary; it will then shutdown all the running processes (as the other server would be active if it were running, which IT MUST NOT BE); and you stop FOE to prevent replication.

                              This puts the secondary servers in a passive state, so it will be consistent with what is about to happen.

                               

                              See post below for a better version of the checklist I once had here

                      • Re: Orion Upgrades with FOE
                        RichardLetts

                        Here is (part of) the checklist I run through during an upgrade. I actaully print this out and put tick boxes when I do a step. (my current checklist has commands and notifications, and any gottchas from the last upgrade)

                         

                        It also includes the running state of the solarwinds processes, if the network can pass traffic, and the FOE state and FOE running state

                         

                        StepNotes

                        FOE Upgrade Instruction

                        (from a now removed Solarwinds document)

                        App1pollerSecondary
                        App
                        secondary
                        poller
                        PrimarySecondaryReplicationApplication running?
                        ApplicationNetworkFOEApplicationNetworkFOE
                        Failover to Secondary1cStoppedBlockedPassive-OnRunningPassActive-onRunningYes
                        Shutdown FOE everywhere, leaving applications running1d,3StoppedBlockedPassive-OnRunningPassActive-onRunningYes
                        Rconsole into all secondary servers

                        Secondary Servers Only

                        2n/an/aStoppedBlockedPassive-OffRunningPassActive-offStoppedYes
                        Check FOE shutdownn/an/aStoppedBlockedPassive-OffRunningPassActive-offStoppedYes
                        upgrade app server4an/an/an/aStoppedBlockedPassive-OffRunningPassActive-offStoppedYes, but unstable
                        upgrade pollers4an/an/an/aStoppedBlockedPassive-OffRunningPassActive-offStoppedYes, but unstable
                        if server reboots login & shutdown FOE4b3n/an/aStoppedBlockedPassive-OffRunningPassActive-offStoppedYes, but unstable
                        Verify application is workingStoppedBlockedPassive-OffRunningPassActive-offStoppedYes
                        Start FOE

                        Secondary Servers Only

                        5n/an/aStoppedBlockedPassive-OffRunningPassActive-onStoppedYes
                        Shutdown FOE & applications6n/an/aStoppedBlockedPassive-OffStoppedPassActive-offStoppedNo
                        Use FOE wizard to set primary active8n/an/aStoppedBlockedPassive-OffStoppedPassPassive-offStoppedNo
                        Start FOE9n/an/aStoppedBlockedPassive-OffStoppedPasspassive-onStoppedNo
                        Shutdown FOE10n/an/aStoppedBlockedPassive-OffStoppedBlockedPassive-offStoppedNo
                        using the remote console app for the next section:StoppedBlockedPassive-OffStoppedBlockedPassive-offStoppedNo
                        Disable the Neverfail Packet Filter Driver

                        Primary Servers Only

                        11an/an/aStoppedopenPassive-OffStoppedBlockedPassive-offStoppedNo
                        upgrade app server11bn/an/an/aRunningopenPassive-OffStoppedBlockedPassive-offStoppedYes, but unstable
                        upgrade pollers11bn/an/an/aRunningopenPassive-OffStoppedBlockedPassive-offStoppedYes, but unstable
                        Enable the Neverfail Packet Filter Driver11cn/an/aRunningBlockedPassive-OffStoppedBlockedPassive-offStoppedYes, but blocked
                        In the FOE config wizard; make primary servers activen/an/aRunningBlockedActive-OffStoppedBlockedPassive-offStoppedYes, but blocked
                        Start FOEn/an/aRunningPassActive-OnStoppedBlockedPassive-offStoppedYes
                        Start FOE on Secondary serversn/an/aRunningPassActive-OnStoppedBlockedPassive-onPossibly runningYes
                        Ensure replication is runningRunningPassActive-OnStoppedBlockedPassive-onRunningYes

                         

                        I ought to note an optimization here that prevents false alerts:

                        11c should be done AFTER you make the primary servers active in the config wizard and and start FOE up, otherwise you have a short time when the pollers do not have network connectivity

                         

                        There are two periods during the upgrade process where BOTH servers have their monitoring network interface in the blocked state

                         

                        EDIT: Note: the step numbers refer to a document no-longer available from Solarwinds; I'm not going to edit this reply and everything else in this thread so after posting my own upgrade procedure documents I'm going to maintain them until we stop using the FoE.