When Patching Can Do More Harm than Good
Conventional wisdom has long stated that patches should be tested before they are deployed. Although some organizations take the concept of patch testing to the extreme, many organizations do not do patch testing at all.
It is difficult to fault smaller organizations for neglecting to test patches. After all, IT departments are usually budget strapped, and busy administrators often lack the time and the resources to do comprehensive patch testing. Even so, blowing off patch testing can have consequences. There are a number of situations in which applying an untested patch can actually harm your systems.
The most widely accepted reason for patch testing is the simple fact that software companies occasionally release buggy patches. Sometimes the bugs within such a patch can break a mission-critical application. For example, back in 2010 Microsoft released a patch (MS10-024) for Windows Server 2008 and 2008 R2. This particular patch broke the Windows SMTP service, which caused certain types of servers to stop sending email messages. This is just one example of a major software vendor publishing a critically flawed patch. This incident is far from isolated. Although it isn’t every day that a major software company releases a patch that breaks a critical service, it does happen often enough to warrant patch testing.
A much less talked about reason why it is important to test your patches is because there have been incidents of patches intentionally disabling or removing application features. Although this scenario might sound implausible, it actually happens on a fairly regular basis. For example, when Microsoft released Service Pack 1 for Exchange Server 2010, there were at least five features that were discontinued (http://technet.microsoft.com/en-us/library/aa998911.aspx#DiscRTMtoSP1).
Microsoft isn’t the only company that sometimes disables or removes features through patches. Application vendors have been known to do the same thing. To give you a more concrete example, consider an app that I run on my Windows smart phone.
The nature of what I do has me on the road on at least a weekly basis. Traveling back and forth to the airport so many times makes it easy to forget where I parked the car. I sometimes catch myself wondering if I am parked in Section A Row 2, or if that was where I parked last week. To make it easier to keep track of my car, I purchased a parking app that keeps track of your car's GPS location. The app worked great for several months until a new patch was released. The patch changed the entire nature of the app. Now, the app leads you to the parking lot, but it won’t show you where your car is located in the parking lot like it used to.
Every once in a while a problem with a patch or with your patch management software might actually cripple an operating system. Over the last few years, I have seen at least a couple of situations in which someone applied a patch only to have the patch installation process crash halfway through, rendering the computer unusable. Although this isn’t a common problem, it does happen.
Another good argument for doing at least a minimal amount of patch testing has to do with the patch deployment process itself. If an organization has a policy of testing patches before they are applied then it stands to reason that the Windows operating system will not be configured to automatically download and install patches at will. One reason why this is important is because the Windows operating system always seems to have impeccable timing when it comes to automated patch deployment.
I have lost count of the number of times that I have been out of town and needed to remotely connect to my computer at home only to find that Windows installed some patches and rebooted the system. Likewise, I have been in the middle of working on a project, gotten tired and gone to bed for the night, only to wake up the next morning and find that Windows was rebooted in the middle of the night as a result of the patch installation process.
As you can see, there are a lot of bad things that can happen if you allow patches to be automatically deployed without doing at least a minimal amount of patch testing. IT pros are busy, and we all know that patch testing takes time. But taking just a few minutes out to make sure that patches deploy correctly and don’t break anything in the process can save you a lot of trouble.