Open for Voting
over 1 year ago

HA upgrades and patches, without loss of service, by using a temporary database

I have to say, the new Orion High Availability (HA) is lovely to use, controlled by an attractive UI, it is quick, intuitive, easy to deploy and use. emoticons_grin.png

However, it doesn't give true HA. emoticons_sad.png

As every time we install a hotfix, patch or upgrade,  we end up with ~50 minutes of loss of service.

All of the patches/upgrades require HA to be stopped and then the active Orion server to be upgraded first.

Then the standby server is patched/upgraded, whilst being kept in standby throughout the upgrade.

The problem being, it seems, is the database, which needs to be upgraded with the primary Orion server.

                                                  Normal operation

So why can't the procedure create a temporary database instance, which will allow the primary node to continue polling, receiving incoming traffic and most importantly alerting, whilst the secondary node is patched/upgraded.

Upgrading the standby server and the database, whilst the active server continues polling and collects the data  into temporary database.

This way the original database can be upgraded, whilst the primary Orion node does it's thing into the temporary database.

Then when the primary database and then secondary Orion server are up to date, make the secondary the primary Orion server and take the over monitoring.

Upgrade the standby server, without interruption to monitoring.

Then once the upgrade is finished on the rest of the Orion servers, the collected data from the temporary database can be processed back into the primary, back filling the gaps in the data.

Process the data from the temporary database, filling in those gaps...

Et viola!

True HA emoticons_grin.png

Now I know this is a very top level thought, to something that no doubt would involve a considerable amount of work, but the result of which would be a true HA solution.

Top Comments

Parents
  • I was under the impression that it did allow for upgrades and patching without downtime.  that is disappointing and unfortunate.

  • Of the operating system, yes. Orion application upgrades require database schema updates which cannot be performed while data is being written to those tables, which is HA must be stopped and APEs shutdown during an Orion upgrade of the main poller. Once the schema is updated, you can upgrade the rest essentially in parallel, but you should avoid bringing APEs online until they've been upgraded, as they could be writing data to the wrong location in the database if that part of the schema has changed.

  • thanks for the clarification aLTeReGo

  • aLTeReGo,

    I know sir that you are The Man, that your knowledge and wisdom is pretty well all knowing, universally recognised and endless, around here.

    Worthy

    But, sir, I must protest!

    I struggle with the idea, that this can't be achieved.

    The upgrade process will be much harder, and would probably require different upgrade packages, and definitely a separate temporary database, with a few temporary SolarWinds services running on it.

    Upgrade that data at the end to mach the new schema, and slot it into the gap (about 50 minutes for me).

    I can understand however the effort of accomplishing this could be quite epic, and the reward apparently so small.

    But you would have a truly enterprise class product at the other end. emoticons_wink.png

  • Thank you yaquaholic​. I wasn't stating that this is an unsolvable problem, or that it's one that we're not interested in solving (I upvoted it myself emoticons_happy.png). I was simply pointing out some of the challenges we face which make this non-trivial. That doesn't mean we aren't making improvements in this area. As you will see when you perform your next upgrade, we've significantly improved the upgrade experience while also dramatically reducing upgrade downtime at the same time. serena​ is aiming to take this to an entirely new level in the future, which is something I'm genuinely excited about and I know you will be to.

  • yaquaholic​ He's not kidding, we have some really cool plans for the future of upgrade. If you're interested, I can add you to a list of user feedback sessions so you can give me more in person feedback. emoticons_happy.png

  • I would love to, and don't get me wrong, coming from FoE to the new HA, has been a delight.

    And the effort that has been put in to the new product, is very obvious. emoticons_happy.png

Comment Children
No Data