Skip navigation

PatchZone

7 Posts authored by: Brien Posey

For years patch management for Windows desktops has been a relatively straightforward process. In corporate environments either a WSUS server or a third party patch management solution downloads patches from Microsoft as the patches become available and then deploys the patches to designated Windows computers. However, this tried and true practice might be changing.

 

With its latest operating system releases, Microsoft seems to have thrown a monkey wrench into the patch management process. My purpose in writing this blog post is not to bash Microsoft, but rather to explain what you can realistically expect with regard to patch management should you choose to deploy Windows 8.

 

Some things stay the same

As you have probably heard, Windows 7 and Windows 8 have a lot in common. In fact, some have even gone so far as to say that when Microsoft created Windows 8 they more or less just bolted the Metro interface on top of Windows 7. In reality, there is more to Windows 8 than just a new GUI. Even so, the similarities between Windows 7 and Windows 8 are undeniable.

 

Given how similar the two operating systems are to one another, it should be no surprise that patch management works more or less the same way in Windows 8 as it did in Windows 7. Windows 8 is able to download operating system patches from a WSUS server or from a third party patch management system. That’s the good news.

 

But Metro complicates it

The bad news is that Metro complicates things. Even though WSUS can be used to patch the core operating system, it is not currently being used to patch Metro apps (which Microsoft says we now have to refer to as Windows Store apps).

 

At first this might not be seem like such a big deal. After all, the Windows Store is used primarily for downloading third-party apps. WSUS has never been used for patching third-party apps. The problem is that from the perspective of patch management, Windows 8 does not really differentiate between third-party Windows Store apps and native applets that are designed to run through the Metro interface.

 

To show you why this is a problem, consider a situation that happened recently. The Windows Store tile indicated that updates were available. When I selected the tile, I expected the updates to be for third-party apps. However, the updates were actually for operating system components (People, Calendar, etc.). In other words, Windows 8 seems to have developed a split brain syndrome when it comes to patch management. The operating system kernel and desktop items seem to still be patched through WSUS, but anything Metro related is being patched (at least for right now) through the Windows Store. This is bound to result in some frustration for Windows administrators.

 

Documentation is also an issue

Another issue is the way that the previously mentioned update was rolled out. Normally, when Microsoft provides an operating system update, there is a corresponding KB article that details what the update does and which files are affected. To the best of my knowledge, no such KB article exists for the December update which updated the Mail, Calendar, People, and messaging apps. Personally, I find the idea of undocumented updates to be a bit unnerving.

 

WSUS does not do Windows RT

Another issue that you need to be aware of with regard to patch management is that currently WSUS is not able to update devices running Windows RT. That isn’t to say that Windows RT devices can’t be updated, it’s just that WSUS is incapable of providing updates to these devices. Rumor has it that WSUS will eventually be retrofitted to support Windows RT, but in the meantime Windows RT updates must come directly from Microsoft Update and from the Windows Store.

    

Conclusion

The fact that Microsoft uses a different update method for Windows Store apps is likely going to prove to be frustrating for network administrators. Hopefully Microsoft will eventually release a new version of WSUS that is capable of providing updates to Windows Store apps and to Windows RT devices.

Windows clusters have always presented a special problem for those tasked with patch management. On one hand, clustering is very necessary. Clustering provides fault tolerance and is often the only way to protect a mission critical server or service. On the other hand, failover clustering compounds the difficulty of patch management.

 

Cluster Patching – The Old Way

Prior to the release of Windows Server 2012, if you wanted to patch a cluster (using only native tools) you had to move the clustered resources off of a cluster node, patch and reboot the cluster node, and then repeat the process for the other nodes in the cluster. The technique worked, but it required manual intervention and was tedious and time consuming. The manual nature of the process could be forgiven if patching was a one time process, but as any good admin knows, patching is an ongoing process.

 

Cluster Patching – The New Way

Those cluster administrators who dread Patch Tuesday will be happy to know that you don’t have to use a manual process to patch a Windows Server 2012 cluster. In fact, Microsoft has introduced a new feature called Cluster Aware Updating.

 

The tricky part of patching a Windows Server 2012 cluster is that cluster aware updating is not used by default. You have to enable cluster aware updating. Otherwise, the cluster will have to be patched using the same manual technique required by clusters running earlier versions of Windows Server.

 

How Does Cluster Aware Patching Work?

Before I explain how to enable cluster aware updating, you might be curious as to how the patching process works in a Windows Server 2012 cluster.  Cluster aware updating works similarly to the method used to manually patch a failover cluster, except that the process is automated.

 

Windows Server 2012 uses a round robin approach to updating cluster nodes. The process begins by identifying the cluster node that has the most free memory available. The clustered resources are moved from a random cluster node to the node that has the most free memory. After the clustered resources have been moved off of the cluster node, the node is placed into maintenance mode. At this point, the cluster node is patched, rebooted, and then taken out of maintenance mode. The process is then repeated for every remaining node in the cluster.

 

How Is Cluster Aware Patching Implemented

Cluster aware updating is based on the use of a new utility called the Cluster Aware Update Tool. The Cluster Aware Update Tool is automatically installed on all of the nodes in the cluster, but it is not active.

 

If you want to use the Cluster Aware Update Tool from outside of the cluster then you can install the tool by installing the Failover Clustering feature on any machine that is running Windows Server 2012 (you don’t actually have to create or join a cluster).

 

In order to perform cluster aware updating, the Cluster Aware Updating tool must run as a clustered server role. That way the update code can move from cluster node to cluster node as the updating process progresses.

 

How To Configure Cluster Aware Patching

To set up cluster aware updating, open the Server Manager and then choose the Cluster Aware Updating option from the Tools menu. When the Cluster Aware Updating tool opens, select your failover cluster from the Connect to a Failover Cluster drop down box, and then click Connect.

 

Once the connection to the cluster has been established, click on Configure Cluster Self Updating Options. This will cause Windows to launch the Configure Self Updating Options Wizard. Click Next to bypass the wizard’s Welcome screen and you should see a message telling you that the cluster isn’t configured with the Cluster Aware Updating cluster role.

 

You must now select the Add the CAU Clustered Role with Self Updating Mode Enabled to this Cluster check box. Click Next and you will be asked to set a self updating schedule. Microsoft recommends scheduling updates to occur at a time when there is the least possible demand on the
cluster nodes.

 

Click Next a couple more times and you will see a screen asking you if you wish to receive recommended updates the same way that you receive important updates. After making your decision, click Next and verify that the information displayed on the summary screen is correct. You can
complete the process by clicking Apply, followed by Close.

 

As you can see, it is relatively easy to implement cluster aware updating. After doing so, the process of keeping the cluster nodes up to date will be completely automated.

What is Flame?

Defending network endpoints against malware tends to be a high priority in almost any organization. Although most of the major anti malware products do a good job, current malware protection is anything but perfect. Perhaps the best evidence of this is flame malware. Flame is an especially sophisticated piece of malware that has been used in targeted attacks in the middle east. There are a number of things that make flame unique, but the most alarming aspect of flame was that it has been confirmed that flame had infected high level systems for several years before it was detected. Flame was also unique in its size. Flame tipped the scales at a whopping 20 MB and consists of an estimated 750,000 lines of code. By way of comparison, most viruses have fewer than 150 lines of code. All of that code allows Flame to record audio, screen captures, keyboard activity, and even Skype conversations.

 

How does Flame work?

Of course these capabilities are not unique in and of themselves. There are legitimate parental control applications that utilize similar capabilities to allow parents to monitor their children’s computer usage. What was unique was the way that Flame managed to work its way into high level computer networks. Flame’s authors reportedly went to great lengths to disguise Flame as a legitimate Content Management System platform. Furthermore, the code was signed using a counterfeit Microsoft digital signature. Flame’s developers found a Microsoft Terminal Server Licensing certificate that had been accidentally authorized for code signing. Because this certificate used a relatively weak MD5 hash, the Flame developers were able to reverse engineer the certificate and use it to digitally sign their own code, giving the illusion that the code had come from Microsoft.

 

Risk of Flame impacting an organization

The odds of an organization becoming infected with Flame are slim. After all, Flame was designed for use against very specific targets. Even so, other malware authors now know that someone succeeded in creating a sophisticated form of malware that appeared to have come from a reputable source and that alluded detection for years. With that in mind, the big question is how organizations can protect themselves from similar, copycat malware that might be discovered in the future and that could potentially already exist.

 

How to protect against malware attacks

The best way to protect against such a sophisticated form of malware is to practice defense in depth. There is no such thing as a perfect security product. None of the antivirus products were able to detect Flame. The only way to guard against this sort of malware is to follow a very rigid set of security best practices.

 

Keeping your systems up to date with the latest patches is a good first step in preventing malware from being able to steal sensitive data. It is equally important however, to use a reputable patch management system. Remember, Flame posed as legitimate Microsoft code. It is therefore conceivable that a future exploit could pose as a Microsoft patch.  As such, it is critically important to use a patch management system that can be trusted to download patches directly from Microsoft. It is also a good idea to get into the habit of cross referencing patch numbers with TechNet to make sure that patches are legitimate prior to deploying them.

 

Another lesson learned from the Flame exploit is to be careful about where you download software from. Patches, drivers, and other updates should be downloaded directly from the vendor, not from a third party Web site. Flame posed as a legitimate CMS platform. While I don’t know where Flame was downloaded from, I am relatively confident that it was not downloaded directly from the Microsoft Web site. Microsoft and most other reputable software vendors go to great lengths to make sure that the code that they make available for download is not infected with malware. If you attempt to download a Microsoft patch from a non Microsoft Web site, there is a very good chance that you will end up downloading malicious code.

 

Although there is no method that is 100% guaranteed to keep malware off of your systems, being careful to download code from reliable sources and scrutinizing applications, drivers, and patches prior to installing them can go a long way toward keeping you safe from infection.

Mobile users have long presented a special challenge to IT professionals when it comes to the task of patch management. The problem is that under normal circumstances centralized patch management only works when the end user is logged into the corporate network. This can be a problem since mobile users might only rarely bring their laptops into the corporate office.

 

In the past, IT pros have used two main techniques to deal with this challenge. One of those techniques involves decentralizing the patch management process. The Windows operating system can be configured to download patches directly from Microsoft and to install them automatically. Many third party applications offer similar functionality.

 

Decentralized patch managment

Decentralizing patch management for mobile users can be very effective because patches are automatically downloaded from the Internet, even if the end user is not connected to the corporate network. The disadvantage to using this technique, however, is that it leaves a lot to chance. Most larger organizations like to test patches before they allow them to be applied to production systems. When you decentralize patch management for mobile users, patches are automatically applied to the user’s laptops whether the organization has tested them or not.

 

Furthermore, this technique makes it difficult for administrators to determine which patches have been applied to mobile user’s laptops. When you decentralized patch management you also forfeit centralized patch deployment reporting.

 

VPN Connections

The other technique that has been traditionally used to handle patch management for mobile users is to deploy patches over VPN connections. In this scenario, the organization maintains centralized control of the patch management process. This allows the administrator to perform patch testing before any patches are authorized for deployment to production systems. When a mobile user logs into the corporate VPN, their computer checks in with the patch management server and downloads any missing patches.

 

On the surface this technique seems ideal, but it has one fatal flaw. The problem is that patch management can only occur when the end user is logged into the corporate VPN. If the end-user only connects long enough to check their email, then it is unlikely that the session will last long enough for the patch management process to complete.

 

Either of these techniques can be used to facilitate patch management for mobile user’s laptops. As you have seen however, neither of these methods is perfect. Even so, there is a third method that directly addresses the shortcomings of the previous two methods.

 

DirectAccess

This method involves using Microsoft’s DirectAccess feature to facilitate patch management. If DirectAccess sounds familiar, it may be because Microsoft introduced it with Windows 7 and Windows Server 2008 R2 as a next generation alternative to traditional VPNs. However, DirectAccess didn’t catch on because it was extremely difficult to configure.

 

Even though the first incarnation of DirectAccess was a flop, the story isn’t over. Microsoft revisited the DirectAccess feature in Windows Server 2012 and Windows 8 and made it much easier to configure. In fact, you can now deploy DirectAccess with only a few mouse clicks.

 

The reason why DirectAccess can be beneficial to mobile patch management is because it eliminates the need for a user to log into a corporate VPN. When a user’s connects to the Internet, Windows automatically establishes a DirectAccess session to the corporate network. This connection is established from behind the scenes with no end user involvement.

 

DirectAccess can be thought of as an always on connection to the corporate network. If the laptop has Internet connectivity, it also has connectivity to the corporate network. This makes it a lot easier to perform centralized patch management because administrators no longer have to worry that end users will only log on for a few short minutes to check their E-mail. Mobile users often remain connected to the Internet for several hours at a time, which is perfect for deploying patches.

 

Requirements to use DirectAccess

In case you are wondering, there are some client side requirements that must be met in order for an end user to establish DirectAccess connectivity. The only client operating system that natively supports DirectAccess is Windows 8; however, there is an update available for Windows 7 that allows it to work with the Windows Server 2012 version of DirectAccess. The client computer will also have to have a TPM chip in order to work with DirectAccess.

 

The Feather in the Cap

When all is said and done, DirectAccess might be the perfect mechanism for facilitating patch management for mobile users. It allows the organization to maintain centralized control of the patch management process, it doesn’t require the end user to do anything special, and the amount of time that users tend to spend online should help the patch deployment process to complete.

 

And because the DirectAccess connection is bi-directional, when combined with SolarWinds Patch Manager it gives you the ability to deploy updates on-demand, as well as perform other systems management tasks.

When Patching Can Do More Harm than Good

 

Conventional wisdom has long stated that patches should be tested before they are deployed. Although some organizations take the concept of patch testing to the extreme, many organizations do not do patch testing at all.

 

It is difficult to fault smaller organizations for neglecting to test patches. After all, IT departments are usually budget strapped, and busy administrators often lack the time and the resources to do comprehensive patch testing. Even so, blowing off patch testing can have consequences. There are a number of situations in which applying an untested patch can actually harm your systems.

 

Bug Introduction

The most widely accepted reason for patch testing is the simple fact that software companies occasionally release buggy patches. Sometimes the bugs within such a patch can break a mission-critical application. For example, back in 2010 Microsoft released a patch (MS10-024) for Windows Server 2008 and 2008 R2. This particular patch broke the Windows SMTP service, which caused certain types of servers to stop sending email messages. This is just one example of a major software vendor publishing a critically flawed patch. This incident is far from isolated. Although it isn’t every day that a major software company releases a patch that breaks a critical service, it does happen often enough to warrant patch testing.

 

Feature Removal

A much less talked about reason why it is important to test your patches is because there have been incidents of patches intentionally disabling or removing application features. Although this scenario might sound implausible, it actually happens on a fairly regular basis. For example, when Microsoft released Service Pack 1 for Exchange Server 2010, there were at least five features that were discontinued (http://technet.microsoft.com/en-us/library/aa998911.aspx#DiscRTMtoSP1).

 

Microsoft isn’t the only company that sometimes disables or removes features through patches. Application vendors have been known to do the same thing. To give you a more concrete example, consider an app that I run on my Windows smart phone.

 

The nature of what I do has me on the road on at least a weekly basis. Traveling back and forth to the airport so many times makes it easy to forget where I parked the car. I sometimes catch myself wondering if I am parked in Section A Row 2, or if that was where I parked last week. To make it easier to keep track of my car, I purchased a parking app that keeps track of your car's GPS location. The app worked great for several months until a new patch was released. The patch changed the entire nature of the app. Now, the app leads you to the parking lot, but it won’t show you where your car is located in the parking lot like it used to.

 

Crippled OS

Every once in a while a problem with a patch or with your patch management software might actually cripple an operating system. Over the last few years, I have seen at least a couple of situations in which someone applied a patch only to have the patch installation process crash halfway through, rendering the computer unusable. Although this isn’t a common problem, it does happen.

 

Bad Timing

Another good argument for doing at least a minimal amount of patch testing has to do with the patch deployment process itself. If an organization has a policy of testing patches before they are applied then it stands to reason that the Windows operating system will not be configured to automatically download and install patches at will. One reason why this is important is because the Windows operating system always seems to have impeccable timing when it comes to automated patch deployment.

 

I have lost count of the number of times that I have been out of town and needed to remotely connect to my computer at home only to find that Windows installed some patches and rebooted the system. Likewise, I have been in the middle of working on a project, gotten tired and gone to bed for the night, only to wake up the next morning and find that Windows was rebooted in the middle of the night as a result of the patch installation process.

 

Conclusion

As you can see, there are a lot of bad things that can happen if you allow patches to be automatically deployed without doing at least a minimal amount of patch testing. IT pros are busy, and we all know that patch testing takes time. But taking just a few minutes out to make sure that patches deploy correctly and don’t break anything in the process can save you a lot of trouble.

The only way to have a truly effective patch management solution is to gain centralized control over the patch management process. One of the challenges standing in the way of achieving this type of control is the fact that many software vendors deploy proprietary patch installers on the user’s desktops. If an organization is ever to fully gain control over the patch management process, the desktop level patch installers must be replaced by a centralized solution. Unfortunately, this can prove to be more difficult than you might expect.

 

If an organization wants to get rid of desktop level patch installers then there are two things that must be done. First, the organization must remove any existing patch installers from the user’s desktops. Second, the organization must take steps to prevent any future desktop level patch installers from being deployed.

 

Cleaning Up Desktop Level Patch Installers

Cleaning up desktop level patch installers can be a very challenging process. In order for you to be able to remove a desktop level patch installer, the installer must exist separately from the application that it is intended to update. Often times if you look in the Programs section of the Control Panel, you will see separate entries for an application and its update mechanism (the patch installer). Assuming that an application and its patch installer are divided into separate components then you can use the Control Panel to simply uninstall the patch installer for the application. Of course this manual removal method is impractical in all but the smallest environments. That being the case, I recommend using a PowerShell script to remove the patch installers. Microsoft provides a very good article on how to do so at: http://blogs.technet.com/b/heyscriptingguy/archive/2011/12/14/use-powershell-to-find-and-uninstall-software.aspx

 

Preventing Patch Installer Deployment

There are a couple of different methods that you can use to prevent desktop patch installers from being deployed. One method involves packaging applications yourself. If you simply deploy applications to the desktop in the usual manner than you will most likely end up deploying the patch installers for those applications as well.

 

As an alternative, you might consider using an application repackaging tool. Such tools typically allow you to deploy an application onto a lab computer, make customizations to the application (such as uninstalling the patch installer that was deployed by default), and then repackaging the application. This approach makes it possible to deploy an application without deploying its patch installer.

 

Conclusion

Depending on the resources that you have available, removing and preventing the installation of desktop patch installers might not always be practical using the methods that I have described. If budgetary or technical limitations stand in your way then you might choose to prevent patch installers from running rather than trying to remove them. Microsoft provides a tool in Windows Server 2008 R2 and Windows Server 2012 called AppLocker. AppLocker allows you to create policies that control what software is and is not allowed to run on desktop PCs.

Editor's Note: PatchZone welcomes our newest guest blogger - Brien Posey.  Brien, a six-time Microsoft MVP, is a published author of over 4,000 technical articles and papers and three dozen book contributions. He has served as a CIO in the healthcare industry and a network administrator for the United States Department of Defense. Check out his blog here.

 

 

Today almost every software vendor routinely releases patches for their wares. Because of the overabundance of available patches, it is critical for organizations to adopt a comprehensive patch management policy. Doing so is the only way to stay organized and to keep track of which patches have been deployed and why. This blog post outlines some important elements that should be included in an organization’s patch management plan.

 

Patch Testing Period

Probably the most hotly debated aspect of patch management is the testing period. Some organizations go way overboard with patch testing, subjecting each patch to a battery of tests for six months or more. On the other end of the spectrum, there are organizations that do not do patch testing at all.

 

The reason why patch testing is such a hotly debated topic has to do with the double edged sword that is patch management. It is a well-documented fact that some patches have bugs. If an organization fails to test their patches then they could experience problems related to a buggy patch. On the other hand, most patches are designed to address security vulnerabilities. An organization that spends an excessive amount of time testing patches leaves themselves vulnerable to the exploit that the patch is designed to correct until the day when the patch is eventually applied.

 

Your patch management policy should define the amount of time that is allowed for patch testing once a patch is released. Some patch management policies define multiple testing schedules based on the patch type. For instance, a patch that addresses a critical vulnerability might be tested over a shorter duration than a more comprehensive patch such as a service pack or a feature pack.

 

Of course determining an acceptable testing schedule is only one aspect of the patch management policy. Typically the policy will also stipulate how the testing is to be done. For example, some organizations use lab testing, while others prefer to perform pilot patch deployments in the production environment.

 

Patch Deployment Method

Another aspect of the patch management process that should be outlined in your policy is the patch deployment method. Typically this is a simple statement outlining the technical aspects of installing the patches and verifying the success of the installation process.

 

Post Deployment Testing

Even with a high degree of patch testing, there is always a chance that a bug could be discovered after a patch has been approved for use in the production environment. As such, some organizations like to do post deployment testing as a way of verifying that the patch is not causing any problems.

 

If your organization does post deployment testing then your patch management policy should outline the testing methods and the testing schedule.

 

Patch Rollback
If a patch is determined to be problematic after it has been deployed to the production network then the patch may need to be removed. Your patch management policy should discuss the technique for removing the patch, but also the situations that warrant removing the patch.

 

Most patch management utilities include a mechanism for removing buggy patches, but it is important to understand that patches are sometimes hierarchical. It may be impossible to remove a patch if subsequent patches have been applied, unless the buggy patch and all subsequent patches are removed. Needless to say this is a somewhat drastic step. That’s why it is so important to outline the circumstances under which it is acceptable to remove a buggy patch.

 

Patch Reconciliation

Finally, a good patch management policy should address the subject of patch reconciliation. If your organization is performing comprehensive patch testing and only deploying approved patches then it stands to reason that you should have a list of the patches that have been approved for deployment.  Patch reconciliation involves comparing this list against an inventory of each computer’s contents to verify that all approved patches have been applied and that no unapproved patches have slipped through the cracks.

 

Your patch management policy should state how often a reconciliation report should be created, as well as how to deal with any discrepancies that may be found within the report.

 

Conclusion

The items that I have outlined in this blog post represent some of the elements that should go into a good patch management policy. However, every organization’s needs are different so it is important to custom tailor your patch management policy to meet your own unique needs.

 

 

Sign up for alerts on PatchZone news & tips here.