As the 2016 year is coming to the end, I've looked back and wow it has been the year of the upgrades for me at my day job. While they have all been successful (some took longer than expected ) , there were bumps, tears and even some screaming to get to the finish line. My team and I are seasoned IT professionals but that didn't stop us from making mistakes and assumptions. What I've learned after doing 5 major upgrades this year is that never assume and always be prepared for the worst and hope for the best.
As you embark on the annual journey of upgrades there are so many factors to look at to make sure that it is successful. While it may seem trivia at times depending on the upgrade you do, it never hurts to go through a basic upgrade run through, like a playbook or if you have a PMO work with a Project Manager. Project Managers can be life savers! But you do not need a Project Manager if you take the time to go through and gather all the information and requirements as part of your planning.
After looking back through all the upgrades I've done this year, I decided to write this post and hope that we can all learn a little something with the lessons we've learned and can be avoided by others.
Let’s get back to basics…
Once we start talking upgrades let's go back to the basics and answer the five “Ws” and get some requirements; WHAT, WHY, WHO, WHERE, and WHEN. Understanding those basic requirements goes a long way. It provides the basic foundation for understanding the situation and what all needs to be done.
WHAT- Let’s first ask what are we upgrading? Is this a server operating system upgrade or an application upgrade? Determining the type of upgrade is vital because this will affect the answers to your other requirements. Once you know what you are upgrading you will need to determine if your current environment can support the upgrade. Depending on what you are upgrading, it can feel like opening a can of worms, as you find you may need to upgrade other systems to make sure it’s compatible with the upgrade you are trying to complete. You may also find out that the upgrade reaches beyond your realm of responsible and crosses over into our departments and function. A “simple” application upgrade can end up costing millions if your current environment does not support all components.
Some examples questions to ask:
- If you're doing an application upgrade does your current hardware specs meet the recommendations for the newer version? If it does not, you may need to invest in new hardware.
- Is this an operating system upgrade?
- Is this an in-place upgrade or parallel environment?
- Or a complete server replacement?
- Can you go direct to the new version or will you need to install CU’s to get current?
- Can your current network infrastructure support the upgrade? Does it require more bandwidth?
- If you are using Load Balancers or Proxy Servers, do those support the upgrade?
- Are there client applications that connect to your systems and are you running supported versions of the client applications?
- Do you have Group Policies that need to be modified?
- What other applications “connect” maybe impacted?
- Are there any legacy customizations in the will be impacted?
- Will there licensing impacts or changes with the upgrade?
Sample upgrade scenario:
An application like Exchange Server has far reaching impacts beyond just the application. If an Exchange DAG is implemented the network must meet certain requirements to satisfy successful replication between the databases across datacenters. Working with your network team ensures your requirements are met. You will may possibly need the storage team if you are using a SAN for your storage which may require new hardware and we all know that can be a project in itself to upgrade a SAN.
An often overlooked item is the client connection to exchange. What version of Outlook are users using to connect to get to their email? If you are using an unsupported version of outlook users may have issues connecting to email. Which we all know would be a nightmare to deal with. Let’s look at the impact of outlook versions to an exchange upgrade. If your outlook versions are not supported, you will need to work with the desktop teams to get everyone to a supported version. This can be costly, from the support to implementing and deploying the upgrade to supported outlook versions and then depending on how your Microsoft Office is licensed you may need to buy additional licenses and we all know that isn’t cheap.
WHY - Let’s ask why are you doing the upgrade? Is the upgrade needed to address an issue or this to say current? Once this has been identified, you can find out what and or if new features you are going to be getting and what value does it bring to the table.
Additional questions to ask:
- Will any new features impact my current environment?
- If I am addressing an issue with the upgrade, what is it fixing and are there other workarounds?
- Will the upgrade break any customizations that maybe in the environment?
- Can the upgrade be deferred?
WHO- Once you’ve figured out the “WHAT” you will know the “WHO” that need to be involved. Getting all the key players will help make sure that you have your ducks in a row.
- What other teams will you need to have involved?
- Network team
- Security Team
- Storage Team
- Database Team
- Server Team
- Application Team
- Desktop Support
- Help Desk
- Project Management Team
- Testing and certification Team
- Communications team to inform end users
- Any other key players – external business partners if your systems are integrated
In certain cases, you may need even need a technology partner to help you do the upgrade. This can get complicated as you will need to determine who is responsible for each part of the upgrade. Having a partner do the upgrade is convenient as they can assume the overall responsibility of the success of the upgrade and you can watch and learn from them. Partners can bring value as they are often “experts” and have done these upgrades before and should know the pitfalls and what to watch out for. If you are using a partner, I would recommend you do your own research in addition to the guidance and support provided by the partner because sometimes the ball can be dropped on their end as well. Keep in mind they are humans and may not know all about a particular application, especially it’s very new.
WHEN- When are you planning to do the upgrade? Most enterprises do not like disruptions so you will need to determine if this must be done on the weekends or can you do the upgrade without impacting users in production during the weekday.
The timing of your upgrade can impact other activities that maybe going on in your network. For example, you probably do not want to be doing an application upgrade like Skype for Business or Exchange the same weekend the Network team is replacing/upgrading the network switches. This could have you barking up trees when there isn’t really need to be.
WHERE– This may seem like an easy question to answer but depending on what your upgrading you may need to make certain arrangements. Let’s say your replacing hardware in the datacenter, you will certainly need someone in the datacenter to be able to perform the switch out. If your datacenter is hosted, perhaps you will need hands on tech to perform a reboot of the physical servers in the event the remote reboot doesn’t work.
I’ve been in situations where the reboot button doesn’t work and the power cord of the server had to be pulled to reboot the server back online, this involved getting someone on in the datacenter to do that. Depending on your setup and processes this may require you to put support tickets in advance and coordination with the hosting datacenter hosting team. Who wants to sit waiting around for several hours to have a server rebooted just to progress to the next step in an upgrade?
HOW - How isn't really a W but it is an important step. Sometimes the HOW can be answered by the WHAT, but sometimes it can't so you must ask "HOW will this get upgraded?". Documenting the exact steps to complete the upgrade, whether it's in place or parallel environment will help you identify any potential issues or key steps that maybe missing from the plan. Once you have the steps outline in detail it's good to do a walk through of the plan with all involved parties so all the expectations are clear and set. This is also helps prevent any scope creep that could appear along the way. Having a documented detailed step plan will also help during the actual upgrade in event something goes wrong and you need to do troubleshooting.
Proper Planning keeps the headaches at bay…
It would seem common sense and almost a standard to answer the 5 W’s when doing upgrades but you would be surprised but how often how many questions are not asked. Too often we get comfortable in our roles and overlook the simple things and make assumptions. Assumptions can lead to tears and headaches if they cause a snag in your upgrade. However, lots of ibuprofen can be avoided if we plan as best as can and go back to the basics of asking the 6 W’s for information gathering.