My Upgrade to Orion 2020.2: A Customer Perspective

With any upgrade to production software, I’m always a little hesitant to be the very first person out of the gate. Like many of you, I’ve been burned in the past when chasing the newest promised features. This isn’t indicative of any specific vendor’s software (SolarWinds included), it’s just being burned is a great way to learn to never do that thing again (the “hot stove” methodology to learning).

The Orion Platform 2020.2 updates were made generally available (GA) on June 4, and I was anxious to try out some of the new features. I used the Orion Demo to review some of the features and see how the products integrated. However, I took my time and did my research before embarking on this last upgrade.

For me, research includes reading release notes, review any hotfix details, cataloging my existing system, reading up on other experiences on THWACK, speaking with friends in the community, and otherwise doing my due diligence. Pretty much the same thing I do with any production system upgrade.

Background

Let me tell you a little bit about my environment, so we’re all on the same page. My SolarWinds infrastructure isn’t the largest nor the smallest build—I like to think of it as the Goldilocks build—just the right size for me and my company. My Orion Platform footprint consists of a Main Polling Engine, two Additional Polling Engines (APEs), and two Additional Web Sites (AWSs). All these servers (and the Microsoft SQL Server hosting the Orion database) are virtualized.

Upgrade Summary

Monitoring Infrastructure

  • Main Polling Engine (MPE)
  • 2 × Additional Polling Engines (APE's)
  • 2 × Additional Web Servers (AWS's)
  • O/S: Windows Server 2016
  • SQL O/S: SQL Server 2016 SP2
  • 5 products on 5 Orion servers

Monitored Elements

  • 1,645 Nodes
  • 12,800 Interfaces
  • 650 Applications
  • 200 Virtual Hosts

Upgrade Plan

  1. Backups and snapshots
  2. Install any prerequisites
  3. Upgrade MPE
  4. Upgrade APE's and AWS's via web
  5. Planned Change Window: 6 hours

Upgrade From/To

  • NPM 12.4 --> skipping 2019.4 --> 2020.2
  • SAM 6.8 --> skipping 2019.4 --> 2020.2
  • NCM 7.9 --> skipping 2019.4 --> 2020.2
  • VMAN 8.4 --> skipping 2019.4 --> 2020.2
  • SCM 1.1 --> skipping 2019.4 --> 2020.2

Total Upgrade Time: 127 minutes
And that includes a opening a support case

This environment currently monitors 1,645 nodes, 12,800 interfaces, 650 applications, 200 virtual hosts, and many other things I’m glossing over for brevity. Our database is currently clocking in at between 70-75 GB. All told, we have 16K network elements, with roughly 550K events spanning our three polling engines.

Planning

Since I was two major revisions behind on most of my products, I decided to add a little extra time in planning. I started by looking at what SolarWinds recommended for my upgrade process by using the Upgrade Advisor.

Upgrade Advisor

I jumped into my customer portal and opened the Upgrade Advisor. I put in the necessary information about my existing system and the target upgrade.

wluther_0-1597767159936.png

After submitting, I was given a step-by-step list of what needed to be done to bring me up to the selected version numbers.

wluther_1-1597767159954.png

For anyone interested, I’ve attached the PDF export of my upgrade advisor, so you can see what I was using as my guide. This made writing up my change management request super easy. I could just attach the PDF.

The summary of the upgrade plan is this:

  1. Upgrade NPM from 12.4 to 2020.2
  2. Upgrade SAM from 6.8 to 2020.2
  3. Upgrade NCM from 7.9 to 2020.2
  4. Upgrade VMAN from 8.4 to 2020.2
  5. Upgrade SCM from 1.1 to 2020.2
  6. Upgrade DPAIM from 11.1.1 to 2020.2
  7. Upgrade the scalability engines using the Centralized Upgrade

Prerequisite

In the release notes for Orion Platform 2019.4 (the version I’m skipping over), it’s mentioned the products now rely on the .NET 4.8 Framework and will check on its presence before upgrading. Knowing none of my servers had this installed, I elected to do an offline installation of the new framework. Thankfully, all of my servers are running the same operating system (Windows Server 2016), so it was only the one download, which I then copied to each of the Orion servers.

Execution

Now it was time for the actual installation. We kicked off right at the beginning of our scheduled change window—midnight.

12:00 a.m.

The first step was to power down all the servers (including the database server) and take snapshots. At my organization, our preference is powered-off snapshots. Database servers are not fans of snapshots while handling data processing, so we err on the side of caution and turn them all off before taking a snapshot.

After the snapshots were captured, we powered up the database server first. Then we powered up the Main Polling Engine, the Additional Polling Engines, and the Additional Web Servers, waiting for each to come online before moving onto the next.

12:15 a.m.

I installed the aforementioned .NET Framework update and rebooted all the Orion Platform machines. After the reboot, I validated they were all recognizing each other, polling was taking place, and the web interface was fully operational. After this confirmation, it was time to start on the Orion Platform products upgrade.

I began the install using the Orion installer I got from my customer portal, and we were off to the races.

The complete install of all six products (not just NPM) took approximately 40 minutes. Considering I was two releases behind for a couple of the products, I was pleasantly surprised with the speed. Could it be faster? Of course, it could. I wish every installation I touch would be faster. Needless to say, I was very pleased with only taking 40 minutes to upgrade all products, as there was a time when a single product would take two or three times as long to upgrade. (More on that below.)

12:55 a.m.

After the product installation, the Configuration Wizard kicked off automatically to build the website, modify the database, and adjust the services. I confirmed the settings from before were in place and we were ready to go.

12:56 a.m.

The Configuration Wizard reported an error and stopped. I’ve been using Orion Platform products for many years, so I looked into some likely places for problems. I looked at the Configuration Wizard log, saw a thing or two I could tweak, and ran it again. It failed again.

1:18 a.m.

Realizing this was an issue outside my knowledge, I did the only correct thing: I contacted Support. From my Customer Portal, I logged in and opened a ticket online, so I could provide all my information in advance. I then called in to the support telephone number and referenced that ticket.

We did a screenshare and a modification or two needed to be made directly to my database. Honestly, I didn’t notice the specifics (I’m only an accidental DBA), but it was definitely something I would not have tried myself.

1:44 a.m.

Together with the support rep, we re-ran the Configuration Wizard and this time it completed. I thanked Alexis and I was able to log in to the web console on the Main Polling Engine.

I went to My Orion Deployment, then the Upgrades and Evaluations tab and ran the updates to all four of my other servers. Not one at a time—in parallel! They completed in under 30 minutes for four servers—including the Configuration Wizard.

2:07 a.m.

My entire infrastructure is now upgraded, and I can mark the change as complete.

My Previous Upgrade Experiences

Before I go on about this new build, I wanted to talk a quick minute about upgrades of the past. It seems like just a couple of releases ago when the new installer was released. I thought things couldn’t get much better. After the release of this consolidated installer, we were able to run all product upgrades in a single maintenance window from day one. Prior to that, well, YIKES! I can remember doing upgrades in the long-ago times, and those were not only unforgiving, but they took multiple maintenance windows to complete.

My Upgrade Experience

This may have very well been the fastest, easiest, and most trouble-free upgrade (and that’s with a Support call) I have ever experienced. The work SolarWinds has done over the last few years really shows with how easy my upgrade process was. I noticed something else: any available hotfixes were already slipstreamed into my install. It couldn’t get much simpler.

First Impressions of 2020.2

One word: fast. The web interface feels so much faster than my previous version. I don’t have raw data to prove this, but I’ve been using Orion Platform products for years, and this feels like it’s trying to win a race.

I haven’t had time to explore all the new features in all six of my products, but I’m sure I’ll get to them in due course. There are specific features important to me (modern dashboards) and important to the company (Orion Maps).

Work Still to Do

Since we have been using SolarWinds products, which has probably been over 15 years now, we’ve introduced some levels of customization. To their credit, SolarWinds has implemented enough features over the years, we’ve been able to do away with a large portion of “custom mods.” We still have a few in-house files/mods we keep, but most of our modifications are just JavaScript additions/tweaks. And, thanks to the wonderful folks sharing their knowledge and experience via THWACK, I’ve been able to put together a few PowerShell scripts to help me manage those mods. Additionally, by monitoring my Orion servers with Server Configuration Monitor, I can fill in the gaps quickly. The product keeps track of all the files we need to touch in the website files (since running the config wizard reverts those changes). I just have to look at the SCM tracking of those files and I know which files need to be changed to re-implement our customizations.

Summary

In under 2½ hours (including opening and working a support case) I was able to upgrade to the Orion Platform 2020.2 product releases for five servers monitoring over 16,000 network elements. SolarWinds has made it simpler, easier, and quicker to do upgrades. I’m anxious to start playing with some of the new features. If you’ve upgraded to 2020.2, what new feature is your favorite?

Parents
  • Hi  

    It's been a while for me but it seems like the .NET 4.8 was included in the installer package.  Check for a reference on that.  Anyone please correct me on that if I'm off.  Best of luck either way!
    -KB

    Late edit: that may have been a Windows update package, sorry.

Comment
  • Hi  

    It's been a while for me but it seems like the .NET 4.8 was included in the installer package.  Check for a reference on that.  Anyone please correct me on that if I'm off.  Best of luck either way!
    -KB

    Late edit: that may have been a Windows update package, sorry.

Children
No Data
Thwack - Symbolize TM, R, and C