6 Replies Latest reply on Sep 10, 2012 3:07 PM by aLTeReGo

    Upgrade advice

    bmills99

      Hello all,

       

      My organization is currently running a moderate sized implementation of Orion NPM, APM.

       

      We are currently running Orion Core 2011.1.2, APM 4.2.0, NPM 10.1.3.

       

      We have a dedicated SQL server, primary polling server, four additional polling engines, and an additional web console server. I am looking to upgrade to the most current versions of NPM and SAM. We poll roughly 16k elements with our current implementation. We can not have the system down any extended length of time and I am trying to determine the best possible upgrade path to take with the least possible amount of down time...

       

      Our SQL server will be upgraded in the near future to Windows 2008 R2, MS SQL 2008... I wanted to upgrade our primary polling server and additional polling engines to Windows 2008 and upgrade RAM to at least 8GB per polling engine. All of our servers currently run Windows 2003 R2 with 4 GB RAM per polling engine (primary included).

       

      For upgrading our polling servers I was thinking of setting up a new server with with the desired OS and hardware (windows 2008 R2 8GB RAM) installing the new Orion software and pointing it to our current Orion DB. This would effectively be our new primary polling server. I would, of course, have all other polling engines turned off at this time, and I would then rename and re-IP the newly upgraded server to the old primary polling server's info...I would then turn on all our old additional polling servers. Then one at at time I would remove the nodes from our additional polling servers, remove that server from production, have the OS and hardware upgraded, place the new Orion software on the server, place the server back in production and reassign all nodes previously assigned to this server...  Would this work? I have questions as the old polling engines may not work with the newly upgraded primary polling engine...

       

      I wanted know if anyone else has had experience with this size upgrade of Orion before and if so, how you accomplished your upgrade to minimize down time. Any one has any thoughts it would be greatly appreciated.

       

      Best regards to all!

      Bryan

        • Re: Upgrade advice
          aLTeReGo

          You are correct that once the primary Orion server has been upgraded the additional pollers will not work. At least not reliably. One option for minimizing down time would be to upgrade your primary poller, then move all nodes to assigned to your additional pollers to the primary while you upgrade the additional pollers. This idea is just crazy enough to work. When the poller is overwhelmed it spaces jobs out over time to compensate. This means that if you are polling for status every 2 minutes, after overloading the poller they may not be polled for status less than 4-6 minutes (just guesstimating here). This should get you up and running quickly (30 minutes or so) and afford you time to upgrade your additional pollers to the appropriate Windows OS version and Orion module versions, splitting the number of nodes on your primary Orion server to each additonal poller as they come online.

           

          Upgrading the OS on your primary Orion instance with no down time will require a little more wiggle room. The way I'd handle it if hardware was available is to install the OS of your choice on new hardware, then install Orion. Then when it's time to switch between them, shutdown the primary server (stop services) and run the configuration wizard on the new box, pointing it to your Orion database. Provided you named the servers the same your new and up to date server should now be your Primary Orion server. That should limit down time to 10-15 min tops. You can even do this in conjunction with the upgrade, leaving the old version running on your original primary server, and running the upgrade procedure with the configuration wizard when you transition. 

            • Re: Upgrade advice
              bmills99

              Many thanks for the reply.

               

              We've had a slight change of plans. We are still going to upgrade our OS and hardware for our Orion implementation but it will be over time.

               

              I am going to upgrade our primary polling server first. I have a server staged and ready to go for this. The new server is running Windows 2008 R2 standard edition with 30 GB RAM. This server is currently ready to go for the upgrade.

               

              My plan is:

               

              1. Shut down all polling servers.

              2. Install new versions of NPM and SAM on primary polling server pointed at our old database server. (NPM 10.3.1, SAM v5).

              3. Bring new primary polling server online.

              4. Assign all nodes to new primary polling server

                   Quick question here:

                        Can I run a script directly on the current Orion database to re-assign all nodes to the primary poller or do I have to use the GUI to do this?

              5. Upgrade each additional polling server - NPM and SAM (NPM 10.3.1, SAM v5).

                   A few questions here:

                        Q: Can the actual upgrade installation be performed on the additional polling servers while my primary polling server is online and actively polling?

                        Q: Can the additional polling servers be upgraded at the same time? I have 4 servers to upgrade and was going to start the upgrade on several servers at once but wanted to make sure running the upgrade on different servers at the same time would not cause any issues...

                        Q: Also, our old implementation was running NPM and APM on all additional polling servers, I understand SAM is supposed to be a standalone product now, so how will this work with installing SAM on all of our additional polling servers? I'm assuming it won't be an issue as all of the polling servers point to the same database as NPM.

              6. Bring additional polling servers online one at a time.

              7. Reassign all nodes to previously assigned polling servers.

              8. Update additional web console. - Same question here as with the additional polling servers - Q: Can this upgrade/installation be done with the primary polling server up and running.

               

              Once all this has been completed. I will go through the process of upgrading the additional polling servers as we receive the new hardware. We will also be getting in a new SQL server which should be a pretty straight forward upgrade per the process provided in your documentation.

               

              Once again, I thank you for the reply to any any information you can provide to the questions listed above will be greatly appreciated.

               

              Best regards!

              B

                • Re: Upgrade advice
                  aLTeReGo

                  4. Assign all nodes to new primary polling server

                       Quick question here:

                            Can I run a script directly on the current Orion database to re-assign all nodes to the primary poller or do I have to use the GUI to do this?

                  A: Yes. You can use the Orion SDK/API to write a script that will assign nodes to a specific polling engine.

                   

                  5. Upgrade each additional polling server - NPM and SAM (NPM 10.3.1, SAM v5).

                       A few questions here:

                            Q: Can the actual upgrade installation be performed on the additional polling servers while my primary polling server is online and actively polling?

                            Q: Can the additional polling servers be upgraded at the same time? I have 4 servers to upgrade and was going to start the upgrade on several servers at once but wanted to make sure running the upgrade on different servers at the same time would not cause any issues...

                            Q: Also, our old implementation was running NPM and APM on all additional polling servers, I understand SAM is supposed to be a standalone product now, so how will this work with installing SAM on all of our additional polling servers? I'm assuming it won't be an issue as all of the polling servers point to the same database as NPM.

                  A: The upgrade of your additional pollers should be run after your primary Orion server has been upgraded and running. There is no need to shutdown the primary server while upgrading your additional pollers. However, be sure that all additional pollers are shutdown and stay offline until they've been upgraded to the same version that's running on the primary server.

                   

                  A: I would strongly recommend against upgrading multiple additional pollers simultaneously since it could cause database locks, installation failures, or worst case could result in database corruption.

                   

                  A: SAM can be installed standalone or integrated with NPM or other Orion modules. Since you are already running an integrated NPM/SAM installation I see no reason to change. The standalone configuration option was added primarily so customers were not required to purchase/run NPM if they only needed/wanted SAM.

                   

                  8. Update additional web console. - Same question here as with the additional polling servers - Q: Can this upgrade/installation be done with the primary polling server up and running.

                  A: Yes.

                    • Re: Upgrade advice
                      bmills99

                      Great! Thanks for the information, I have completed my implementation place with the exception of one small detail. In the past when I have upgraded Orion I have had to upgrade APM first then upgrade NPM per Solarwinds tech support. Is this still the case?

                       

                      Best regards,

                      B