Skip navigation

Patch Planning: Steps for an Effective Patch Management Process - Timescales, testing and communication

 

[Editor's Note: PatchZone welcomes Alan Wenlock. Alan is from the UK and works in the Security and AntiVirus field. He's been working with SMS and Configuration Manager for ten years, and is also a moderator at MyITForum.]

 

Patching is a topic that always sends people into a panic for one reason or another. Whether you’re a Service Manager, Technician or Helpdesk Operator the word Patch always brings uncertainty or nervousness.  Within the Patching area it’s not always clear to people how critical the task is, unless you get hit by a known exploit, or how many issues you might encounter as a result of patching.

 

Once of the main questions when looking at patching should be "What is the impact if your system is taken down by a known vulnerability and offline for x hours". 

 

It can be hard work to start implementing a patch process and there will always be opposition against it but once you have a process setup and running it’s amazing how slick you can get your patching process to be.

 

There are several steps to an Effective Patch Management Process, but this is not to say that these are the only steps, and a patch process from one organisation can be totally different from another organisation as it has to adapt and work around the systems you are supporting.  If patching was easy then we'd all have the patches downloading and installing automatically and we know this is not the case so how do you plan your patching process

 

1) Timetable:

It's key to have at least 2 processes when you are looking at Patch Management planning: 

 

a) The first process is going to be for a regular patch cycle, like for example Microsoft’s monthly patch cycle or Adobe's quarterly one.  This process will have a set timetable of activities that you following each month/quarter.  In my experience its always better to try and patch sooner rather than later so in this timeline we’d be typically looking at a week long process giving you time to complete your activities and have sufficient testing. 

 

b) The second process is how you deal with out of sync patches that do not fall under your regular patch activity.  What happens when, for example, Microsoft release a critical patch at the end of the month and the vulnerabilities are being actively exploited.  You can't wait for your regular patch process to come around, although it's an option, so what happens in these cases.  In the case of out of band updates you need to act quicker and I’d always try and get the patch tested and deployed within a 2-3 days to ensure that your endpoints are patched and not vulnerable to the exploit.  In my experience it’s critical to protect as many of your endpoints as possible to start to reduce the risk to your client network(s).

 

2) Testing

This is one of those areas when you need to decide how much testing you do as part of your testing cycle.  It's impossible to test all applications unless you are lucky enough to only have a small number to look after.  With the two different timetables we mentioned above its clear that the out of sync timetable is going to have less testing time then the regular cycle, perhaps a day’s testing at best.  It's important to test the patches on a set of test machines first to make sure there are no issues actually apply the updates or after the reboot.  Once these basic tests are complete you need to involve your apps team.  It may be that your required to test the apps rather than a specific team but it’s important to have some kind of document to show the tests that are going to be done.  This then becomes your baseline for testing each month and helps to maintain a standard patch testing process. 

 

How long you test the patches depends on how critical a system is but also you have to weigh up how critical the patch is.  If the patch is released out of cycle then I’d usually consider only doing a day’s testing, whereas with a regular monthly patch you might allow up to a week to test. As I mentioned earlier every organisation is different but you will still need some guide to how long your testing phases will be.

 

3) Communication

With any process that involves more than one team you are going to need to ensure that communication is taking place and even if you are testing the patches yourself against your applications you still need to make sure people like your helpdesk operators and service managers are kept up to date on any patch rollouts and even your customers so they know when to expect a patch.  This is a key area that doesn’t always get as much attention as it should and I’ve always found that this is an area that can help your patch process run more smoothly.  Customers will always be the first to complain about patches but once you have a regular patch cycle and associated communications they will soon learn to expect it.

 

So we've covered the timetables, testing and the communication at a high level but we have really only touched on the initial few areas of the patch process so far and we still need to look at doing the actual patch rollout and then how to monitor the rollout, liaise with your helpdesk and deal with issues and the potential to have to back out a patch, but that's all for another blog or two.

Thinking about migrating to ConfigMgr 2012?  If so, you should register for this webcast to hear some of the lessons learned over the last 5 years of use with ConfigMgr 2007 with Matt Hudson & Chris Nackers, both SCCM MVPs.  This webcast will cover what problems plague all environments, how some problems are solved with 2012 and how some will never change. This will be an open discussion where we can talk about how the community has solved or failed at solving various problems with ConfigMgr.   

 

Register here: http://bit.ly/ULdXiw

 

Have any burning questions?  Please list questions in the comments section of this post, or ask them during the presentation.

c, It’s patching time; Microsoft released Patch Tuesday updates for September yesterday. Microsoft released only 2 updates and none of them marked critical.  However, we should see an Microsoft Internet explorer soon, as they are working to incorporate the latest Flash update. Per a Patchmanagement.org post, Microsoft is working closely with Adobe to align release schedules as close as possible to ensure Flash Player in Windows 8 and IE10 are always secure.    Updates are light all-around, compared to August.  This month we have seen 3 updates to date from 3rd party application ISVs – Chrome, Firefox and Thunderbird.

 

The two Patch Tuesday updates address four issues in Visual Studio Team Foundation Server 2010 SP1, or Systems Management Server 2003 SP3 or System Center Configuration Manager 2007 SP2.

 


MS12-061 Vulnerability in Visual Studio Team Foundation Server Could Allow Elevation of Privilege

The security update resolves a cross-site scripting (XSS) vulnerability in Visual Studio Team Foundation Server that allows an attacker to inject a malicious script when user visits a specially crafted page using TFS Web access. The script on successful execution can give the attacker elevated privilege access to the user system.

 

 

MS12-062 Vulnerability in System Center Configuration Manager Could Allow Elevation of Privilege

This security update resolves vulnerability in Microsoft System Center Configuration Manager that allows elevation of privilege when the user visits a specially crafted URL. The attacker can pursue or force the users to click the URL, so users need to make sure they don’t click on malicious links.

 

Microsoft Security Advisories

       

Microsoft released the above five security advisories for the month of September 2012, of which the update on ActiveX kill bits seems quite essential. The security advisory sets the kill bits for the following third-party software: Cisco Secure Desktop, Cisco Hostscan and Cisco AnyConnect Secure Mobility Client.

 

Click here to assess your environment against known vulnerabilities for which there is a patch – free for 30 days.

The only way to have a truly effective patch management solution is to gain centralized control over the patch management process. One of the challenges standing in the way of achieving this type of control is the fact that many software vendors deploy proprietary patch installers on the user’s desktops. If an organization is ever to fully gain control over the patch management process, the desktop level patch installers must be replaced by a centralized solution. Unfortunately, this can prove to be more difficult than you might expect.

 

If an organization wants to get rid of desktop level patch installers then there are two things that must be done. First, the organization must remove any existing patch installers from the user’s desktops. Second, the organization must take steps to prevent any future desktop level patch installers from being deployed.

 

Cleaning Up Desktop Level Patch Installers

Cleaning up desktop level patch installers can be a very challenging process. In order for you to be able to remove a desktop level patch installer, the installer must exist separately from the application that it is intended to update. Often times if you look in the Programs section of the Control Panel, you will see separate entries for an application and its update mechanism (the patch installer). Assuming that an application and its patch installer are divided into separate components then you can use the Control Panel to simply uninstall the patch installer for the application. Of course this manual removal method is impractical in all but the smallest environments. That being the case, I recommend using a PowerShell script to remove the patch installers. Microsoft provides a very good article on how to do so at: http://blogs.technet.com/b/heyscriptingguy/archive/2011/12/14/use-powershell-to-find-and-uninstall-software.aspx

 

Preventing Patch Installer Deployment

There are a couple of different methods that you can use to prevent desktop patch installers from being deployed. One method involves packaging applications yourself. If you simply deploy applications to the desktop in the usual manner than you will most likely end up deploying the patch installers for those applications as well.

 

As an alternative, you might consider using an application repackaging tool. Such tools typically allow you to deploy an application onto a lab computer, make customizations to the application (such as uninstalling the patch installer that was deployed by default), and then repackaging the application. This approach makes it possible to deploy an application without deploying its patch installer.

 

Conclusion

Depending on the resources that you have available, removing and preventing the installation of desktop patch installers might not always be practical using the methods that I have described. If budgetary or technical limitations stand in your way then you might choose to prevent patch installers from running rather than trying to remove them. Microsoft provides a tool in Windows Server 2008 R2 and Windows Server 2012 called AppLocker. AppLocker allows you to create policies that control what software is and is not allowed to run on desktop PCs.

You’re sitting at your desk, punching out a few keys on the keyboard, and then it happens.  The moment everyone knows is possible, but always thinks it happens to someone else.  Your phone rings and the person on the other end is now informing you that the company has been part of a data breach.  Your company’s databases and access has been found on an underground website where this type of information is sold and traded.  After hanging the phone up you say to yourself, “How?”, “What about our firewall?”, and “Our monitoring and intrusion prevention should have caught this.”.  These are just some of the questions which begin to flow from you and others.  All these questions are important but at this minute the only one to consider is “How?”.  You need to begin Incident Response procedures and find the cause.

 

Patch Applications and the Systems they run on

The cause was covert, unexpected, but used by many.  A 3rd party application was the open door, the way the attacker found access.  When patching systems many people focus on the system itself but overlook the applications actually installed on these systems.  Attackers are still using the systems, so without a doubt this is important, but the initial entrance and pivot point is hardly the system.  Many times the attacker uses a form of Social-Engineering to gain the users trust, allowing the user to give access to the attacker from within the company; bypassing all the external protections put in place.  This type of attack is often made easier by tools such as TrustedSec’s very own Social-Engineer Toolkit (SET) (https://www.trustedsec.com/downloads/social-engineer-toolkit/ ).   When focusing on patching we all need to take a look at the applications themselves.  I would like to say either the system or the applications is most important; however, I believe they are equally important.  The installed applications need to be documented with versions, by system.  This needs to be maintained frequently and the updates pushed out after following the “5 Common Sense Tips.”

 

Application attacks are a real security threat

Attacks on applications are common, and a real security threat.  When the 3rd party applications are ignored as a threat or simply unmaintained you will not be able to know whether the recently exploited application is in your environment.  That single application, installed just on one system, just for one project, and then forgotten about, has now opened the door and allowed an attacker to exploit your company.  This exploited machine was the pivot point for attacks on every other system, in turn exploiting other applications, and finally gaining access to the databases.  Now your company is being traded in the underground like stocks traded on the world’s largest stock markets.  This breach has caused your company financial losses in such areas as intellectual property, reputation, and IT resources to say the least; all because 3rd party applications were overlooked in the overall security posture of the organization.

 

Document and Patch 3rd party applications

When developing your patching process you need to have a procedure in place for documenting 3rd party applications as well as patching them.  The patch cycle for 3rd party applications are not as defined as Microsoft, so staying vigilant with these vendors is a must.  A great starting point with 3rd party applications can be found at the Table of 3rd Party Patches.  Also consider your patch management solution, consider whether or not it includes support for the Common Vulnerabilities and Exposures (CVE) (http://cve.mitre.org/ ) databases.  Also consider becoming part of the mailing lists at the United States Computer Emergency Readiness Team (US-CERT) (http://www.us-cert.gov/) or a CERT in your region.  Maybe try using resources found at the National Vulnerability Database (NVD) (http://nvd.nist.gov/ ) website. 

 

There are many resources, but ultimately it falls back on making sure the procedure is developed and followed.  Everyone will be hacked at some point, which is a given fact.  The variable in the equation is to what extent are the damages and whether or not you were able to catch the attacker before got everything; otherwise you may be the next person getting that call, having no idea you even had an attacker running through your 3rd party applications.

jkuvlesk

Patch Management News

Posted by jkuvlesk Employee Sep 4, 2012

These are some interesting articles we have come across recently.

 

Java Menace

A new zero-day exploit found in Java causes attackers to exploit a vulnerability that can reveal user data. The vulnerability affects all Java Run-time versions.  Since the exploit has been made public users have no but to disable Java software on their machines or patch Java with the update released on 8/30. 

 

Check out how to patch Java error free.

 

The Rise of the “Blackhole” Exploit Kit: The Importance of Keeping Software up to Date

This article, written by Tim Rains of Microsoft, depicts the increase of reported blocks of HTML & JavaScript exploits was due to the emergence of JS/Blacole.  Blacole is a collection of web pages that contain exploits for vulnerabilities in JRE, Adobe Flash & Reader, MDAC and other products.  Often, these exploits target vulnerabilities that are years old.  Hence, the importance of keeping software up to date.

A related article from Security Affairs

 

Cuts in budget may have negative impact on security

A recent survey from InfoSecurity Europe and Tufin Technologies found 48% of companies focus on cost savings at the expense of security.  Sadly, a lack of endpoint security investment can be both dangerous and costly.

 

ConfigMgr (SCCM) – Troubleshooting Tips to Resolve Scan issues in Software Updates

In this blog post, Anoop C Nair walks through the cause and resolution of a software update SCAN error in SCCM, using an example with Windows 2008 R2 core servers.

 

ConfigMgr 2012 product patch servicing model

This post, by Rob Marshall, summarizes key aspects of the new cumulative update servicing model for System Center Configuration Manager 2012.  Some of the key points:
• SCCM will have patches packed together as single SFX bundle that can be read only from server site
• SCCM bundles will have Server and Client patch MSI’s, license files, scripts and files needed for server to deploy patches
• SCCM patches will be distinct for Server, Client and admin console
• Updates Publisher 2011 is the supported tool of choice for importing patches into ConfigMgr.

 

And one more…..
A better guide to setting up SCUP with a Microsoft PKI