Showing results for 
Search instead for 
Did you mean: 
Create Post

The Software Arms Race: Do You Even Upgrade?

Level 9

After you’ve installed your new storage systems and migrated your data onto them, life slows down a bit. Freshly installed systems shouldn’t throw any hardware errors in the first stages of their lifecycle, apart from a drive that doesn’t fully realize it’s DOA. Software should be up to date. Maybe you’ll spend a bit more time to fully integrate the systems into your documentation and peripheral systems. Or deal with some of the migration aftermath, where new volumes were sized too small. But otherwise, it should be “business as usual.”

That doesn’t mean you can lie back and fall asleep. Storage vendors release new software versions periodically. The interval used to be a couple of releases a year, apart from the new platforms that might have a few extra patches to iron out the early difficulties. But with the AGILE mindset of developers, and the constant drive to squash bugs and add new features, software is now often released monthly. So, should you upgrade or not?

If It Ain’t Broke…

One camp will go to great lengths to avoid upgrading storage system software. While the theory of “if it ain’t broke, don’t fix it!” has its merits up to some point, it usually comes from fear. Fear that a software upgrade will go wrong and break something. Let’s be honest though: over time, the gap between your (old) software version and the newer software only becomes bigger. If you don’t feel comfortable with an upgrade path from 4.2.0 to 4.2.3, how does an upgrade path from 4.2.0 to 5.0.1 make you feel? Especially if your system shows you an uptime of 800+ days?

On the other hand, there’s no need to rush either. Vendors perform some degree of QA testing on their software, but it's usually a safe move to wait 30-90 days before applying new software to your critical production systems. Try it on a less critical system first, or let the new installs in the field flush out some additional bugs that slipped through the net. Code releases have been revoked more than once, and you don’t want to be hitting any new bugs while patching old bugs.

Target and latest revisions

Any respectable storage vendor should at the very least have a release matrix that shows release dates, software versions, adoption rates, and the suggested target release. This information can help you balance “latest features and bugfixes” versus “a few more new bugs that hurt more than the previous fixes.”

Again, don’t be lazy and hide behind the target release matrix. Once a new release comes out, check the release notes to see if anything in it applies to your environment. Sometimes it does really make sense to upgrade immediately, like with critical security or stability patches. Often, the system will check for the latest software release and show some sort of alert. In the last couple of months, I’ve seen patches for premature SSD media wear, overheating power supplies that can set fire to your DC, and a boatload of critical security patches. If you keep up-to-date with code and release notes, it doesn’t even take that much time to scroll through the latest fixes and feature additions.

One step up, there’s also vendors that look beyond a simple release matrix. They will look at your specific system and configuration, and select the ideal release and hotfixes for your setup. All this will be based on a bunch of data they collect from their systems at customers around the globe. And if you fall behind in upgrades and need intermediate updates, they will even select the ideal intermediate upgrades, blacklisting the ones that don’t fit your environment.

How often do you upgrade your storage systems? And what’s your biggest challenge with these upgrades? Let me know in the comments below!

Level 13

like many things in IT it depends. If you're waiting for a new feature or bug fix then upgrade. Equally if it's a security fix then upgrade. There is the old adage "the closer you are to the cutting edge the more likely you are to bleed". So I'd also follow your suggestion of a 30-90 day wait before implementation - just to let any unintentional bugs get fixed first.

We usually have to be looking for a specific feature or fix.

There's no question.  Even if it isn't broken (to the best of your awareness), things need to be upgraded.  For various reasons:

  • Vulnerability Reports released by Cisco or some security team/agency may reveal problems of which you were unaware.  Yes, your systems may be working flawlessly, but you are still required to remediate those vulnerabilities.  Maybe, like me, you are assigned a security/vulnerability score on a monthly basis (all teams in my organization get this score for their devices/systems), and scores improve each time you close a problem.  Maybe your score reflects your abilities as a partner or provider to your customers or community.  Having a good score, having fewer vulnerabilities (or none?) makes you a better organization with which users may choose to do business.
  • Technologies change, and this drives firmware and software upgrades.  Just look at the differences between snmp-v1 and v2c.  Getting the benefits of v2c required an IOS update and a reboot.  And the same for moving up to snmp-v3.  It could all run on the same hardware, but the IOS had to be replaced to get those improved features.   Keeping your gear up to date ensures you can leverage what's available to fix problems and provide better service to your customers.  Not being able to leverage the new tools and procedures that are good, that may become industry-recommended or industry-standards, limits your viability as a partner.  Who'd come to work for you?  Who'd do business with you if you aren't compliant with the standards.  PCI, PHI, SOX, etc. all must have their needs addressed; if it requires upgrading your IOS or firmware or ROMMON, that's what you need to do.  If you have to upgrade your hardware, it's not inexpensive, but it must be done.  I let the management/finance/legal types wrangle over making the books balance properly and legally, with depreciation and amortization over time.  And I let the vendors take it on the chin when their products must be replaced via forklift upgrades when they promise compatibility and we prove their promises fell through.
Level 20

Seems to ring a bell with upgrade NPM from 11.5.x to 12.2 if I'm not mistaken lol!  That wasn't an easy upgrade... it required swapping out the SQL Server as well o.O!

That's my policy with Solarwinds major Release Candidate upgrades.  Once Hot Fix 5 is out there, things are probably pretty stable.  I went through four months of pain with the last upgrade when I was right on top of it.  That's what I get for being anxious and swallowing the Kool-Aid.

Fool me once . . .

Level 9

I didn't know that adage; mind if I steal it? Thanks for commenting!

Level 9

So what would you do with periodic upgrades that don't add fixes or features that you specifically need?

Level 13

Feel free. it's what my old boss used to quote to me back when I first started in IT back in the 80's.


Nice write up

Level 14

Anyone who applies a patch or upgrade asap should not be working in IT.  Vendors often rush out software and don't fully test.  Quite often they can't test every environment.  I read the docs. and watch the internet for any horror stories.  Once I am happy I will plan the upgrade / update.  If it is a fix for an issue we have I will do it within a week or two.  If it isn't I might wait a month or two.  I have always found it best to stay away from the edge in case I fall off.  I still stay reasonably near the edge to enjoy the view.  Remember, It's not the falling off that kills you, it's the sudden stop at the bottom.

Level 13

Concur -- nice write up.

About the Author
Based out of the south of the Netherlands, working with everything from datacenters to operating systems. I specialize in storage, back-up & server virtualization systems and will blog about experiences in the field and industry developments. Certified EMCIEe, EMCTAe and EMCCAe on a number of Dell EMC products and constantly trying to find the time to diversify that list. When not tinkering with IT you can find me on a snowboard, motorbike or practicing a variety of sports.