Skip navigation

Recent changes to the Microsoft certificate landscape will have a significant impact on the processes related to local publishing and third-party updates for customers using WSUS for these type of updates -- all complements of the Flame malware discovered in May.
 
First, KB2718704 (http://technet.microsoft.com/en-us/security/advisory/2718704) was released on June 3rd to revoke three Microsoft certificates that were identified as compromised as a result of the Flame investigation, and is discussed in detail in these articles:


http://blogs.technet.com/b/msrc/archive/2012/06/03/microsoft-releases-security-advisory-2718704.aspx
http://blogs.technet.com/b/msrc/archive/2012/06/04/security-advisory-2718704-update-to-phased-mitigation-strategy.aspx
http://blogs.technet.com/b/msrc/archive/2012/06/06/security-advisory-2718704-collision-attack-details-wu-update-rollout.aspx

Second, KB2720211 (http://support.microsoft.com/kb/2720211) provided an update to the WSUS server which rolled up a couple of previous WSUS hotfixes, one of which affected local publishing, but more significantly, updated the certificate infrastructure for WSUS, and provided a new Windows Update Agent (v7.6.7600.256) which has enhanced security regarding how it trusts certificates for selecting and installing Windows updates.

Third, KB2661254 will be released on Patch Tuesday, August 14, 2012, and will invalidate all pre-existing certificates that use key-lengths of less than 1024 bits. This will include ALL WSUS self-signed local publishing certificates that were created prior to installing KB2720211.

As a result of these updates you should immediately plan for the following:
1. Ensure KB2718704 is installed to all systems.
2. Ensure KB2720211 is installed to all of your WSUS servers. (Note: This update is not trivial. Please refer to the KB article, the TechNet WSUS Forum, and the related blog post for guidance.)
3. Once KB2720211 has been successfully installed to all WSUS servers, you will need to create a *NEW* WSUS self-signed publishing certificate. This action will create a new 2048-bit certificate.
4. Distribute this certificate to your client systems.

Then consider your existing published third-party updates in your WSUS infrastructure. Those updates will continue to be installable by your client systems until such time as KB2661254 is installed. You should make an effort to expedite the deployment of these updates to wherever they may be needed. The installation of KB2661254 will invalidate the previous WSUS self-signed publishing certificates and render all of your existing third-party updates as uninstallable.

For those updates you no longer need to deploy, you should use whatever tools available to you to expire or delete those updates. At a minimum you should decline those updates to ensure they are not accessible to client systems.

For those updates that you still need to have available, you should expire those updates and then wait 24 hours to ensure all client systems have received that change in approval status from the WSUS server. If you have downstream WSUS servers, you may wish to wait 48 hours. Following the expiration of those updates, you can then delete them from your WSUS servers. If you do not have the ability to expire or delete updates, at a minimum you should decline them.

Then, republish any of those updates using the newly created (post-KB2720211) 2048-bit WSUS self-signed publishing certificate.

Editor’s note:  PatchZone welcomes Robert Miller who will be writing a series of blog posts on PatchZone.  Robert has over 14 years of experience with network, system, telephony administration and corporate security and is currently a Security & Operations Manager for a privately held company.  Robert is also a respected security researcher.  Check out his personal blog at www.armoredpackets.com.

 

The Patch Management Process: 5 Common Sense Tips

 

When an IT professional says “We need to update the servers this weekend”, what comes to mind?

 

Many of us simply say to ourselves “Great, another weekend spent in the datacenter…”  However, few never really think about what the patching of a system or application really does; or what is the best approach to take when getting ready to deploy these updates.  Just like many things we as IT professionals do, it is not the action that counts, but rather the prep work that makes a successful deployment, or a dreaded failure.

 

So how can you put together a patch management solutions that will not only give back your weekends, be able to provide the security needed for your organization, but also prevent that dreaded failure?

 

Let’s take a look at the key areas to achieve these goals.

 

I. System Scope

II. Development / Testing Environment

III. Change Management / Change Control

IV. Patch / Update Deployment

V. System Monitoring & Vulnerability Scanning

 

One might think the process begins on the update release date.  This is not the case by any means.  The work really begins even before the very first system is built and configured.

 

I. System Scope
The system administrator, manager, director, and chief information officer need to sit down and outline the purpose of the systems.  This will include the system role as well as the baseline the system will take when first built.


What will the role be of the system?
What applications will run on each system?
What systems are mission critical, mission sensitive, business critical, general operations, and testing?


Once the system scope has been developed and documented you can move on to the next step in the process.  Until you have the scope of what the systems are going to be doing and what category that system resides in, you cannot truly have a successful patch management process.

 

II. Development / Testing Environment

When building your test environment consider 3 key characteristics – is the environment flexibility, dynamic, and accurate?  Flexibility refers to the ability of the environment to serve multiple roles.  Dynamic in the sense that the system can be changed to a different role and do so quickly. You have the flexibility of a system that can serve more than one role but if you have to spend days building it then you are not very dynamic.  Most importantly, in my opinion, is the accuracy of the system as it equates to the production system.  If it does not have the same baseline and installed applications you will not be able to test the patches properly to find issues before rolling it out into the production environment.

 

How can we achieve all three requirements?

 

Virtualize your test environment; create a clone of the physical production system.  Then prior to installing the patch take a snapshot; this way if the patch hoses something up you can simply restore from the snapshot and continue working on solutions.  Virtualization has become so vital in a testing environment; the cost verse reward is incredible.

 

III. Change Management / Change Control

Now incorporate change management.  The key is to make sure you have a process in place prior to installing the patch and even more important is the back out plan for that patch if something goes wrong.

 

IV. Patch / Update Deployment

Let’s move to actually installing the patch or application update.  During this step you need to take good notes, note anything that was required as a dependency which was not documented by the vendor.  While taking these notes you need to observe adverse reactions to the patch which may affect the production environment.

 

The rollout is the last piece of the patch deployment process.  Prior to this step you should already have a confirmed maintenance and patching schedule which will be incorporate not only what time and day of week you deploy patches but also the order of the systems based on system and security scope.


However, the process is yet to be completed!

 

V. System Monitoring & Vulnerability Scanning

Finally you need to run the systems and monitor the operation, as well as use a vulnerability scanner such as OpenVAS or Nessus to verify the systems received the patch or update successfully as well as actually fixed the issue the patch or update was supposed to resolve.


This was a brief example on the steps required in order to build a successful program.  However, understand that every organization will be different and so long as you have the methodology and support you can have a successful patch management program.

Every week, it seems, system administrators ask me how to roll back, or uninstall, patches that have gone wrong.

 

I tell them that if they’re even thinking about rollback before installing a patch, they’re asking the wrong question. Patch rollback is so difficult, risky and time-consuming you should avoid it at almost all costs. Instead, put in your time and effort up front so you can understand which patches are safe enough and important enough to deploy before pulling the trigger. In the long run, this means less work, less trouble and less risk.

 

Don’t believe me? Keep reading.

 

Rollback: Mission Impossible

 

For one thing, Microsoft doesn’t even provide a way to automate the mass rollback of operating system patches. That means writing scripts, which is time and effort you can’t afford.  Even worse, it might mean tracking down which devices you patched and roll them back manually. Then there’s the challenge of restoring any data that was lost due to resulting system crashes, and assuring you’ve found a “known good state” to restore the system to. Let’s not forget the time, and personal pain, involved in explaining to your users why this happened and how you’ll prevent it in the future.

 

Don’t get me wrong: Users have every right to be concerned about the quality of patches, which unfortunately is falling (at least in the Microsoft world.) But all the complications that come with packaging and deploying patches, such as discovering affected systems and tailoring the un-install for each variety of system configuration, also hold true for rolling back the patches. In fact, discovering and “reverse-engineering” the patch process is even more difficult since the failed patch may itself give inaccurate information about which systems have and haven’t been patched.

 

One scenario is that the patch successfully installed, and simply broke something. This would seem to be a simple fix – uninstall the patch, assuming that uninstalling the patch actually does restore all of the previous configurations. A more onerous scenario is that the patch installation is only partially successful. In that case, surgically extracting that patch may become even more tedious, and other things may get further damaged in the process. Depending on the extent to which the patch was installed, the WUAgent may report it as “Not Installed”, even though enough of it was installed to break something else.

 

The answer to the patch quality issue is not to try to build an escape hatch for yourself. It is to instead properly test patches before deployment, leverage patch tests done by others, as shared in such forums as PatchManagement.org., or simply skip patches for less critical systems or less common threats.

 

Test and Prioritize

 

I can hear you now: Who has time to test the dozens of patches I get every month? You can get some level of assurance from commercial packages such as Patch Manager, which test patches to ensure they detect the systems on which they should be installed and that the installer is properly initiated where it should be.

 

You can also use snapshots (virtualization capability) to test patches. Take a snapshot of each virtual machine before applying a patch, or patches. If any issues occur after patching, simply apply the snapshot to restore the machine to the pre-patch state and decide if the patch is important enough to find another way to install it. This can be an effective way to identify a single problematic patch out of a larger group with no risk to production systems.

 

Another common-sense technique is to deploy patches in a pre-production environment before trusting them on your production systems. This is usually a much more limited, and thus manageable, environment in which to track variations among systems and the “before” and “after” state of patched systems. The downside is that, not being a true production system, you may not catch all the problems a bad patch can cause.

 

Another easy way to avoid, or at least reduce, patch problems is to prioritize which patches you deploy. This doesn’t have to be a long and expensive process. Many admins have rough rules such as deploying “only critical security patches, and only on mission-critical systems.” Avoiding “functional” (rather than security) patches may not be elegant, but it’s a good way to beat back the majority of serious threats without wasting your time on optional patches.

 

Bottom Line: Don’t waste time chasing a rollback option that really isn’t an option. Instead, invest some effort up-front in testing and prioritizing your patches to make sure your patches don’t need to be undone.

Microsoft Patches
It’s going to be a busy week for patch administrators as Microsoft released their patch Tuesday updates for July 2012. Microsoft has released 9 bulletins of which 3 are rated critical, with the remaining updates noted as important, resolving a total of 16 vulnerabilities.


Critical Patches
Vulnerability in Microsoft XML Core Services Could Allow Remote Code Execution – This is the most critical update on this patch Tuesday as it resolves the vulnerability in XML Core Services 3.0, 4.0, 5.0 affecting Microsoft Windows, Office and Server Software.
Cumulative Security Update for Internet Explorer– This is a critical update for Internet Explorer which patches the exploit, from which an attacker could execute remote code if the user visits a specially crafted site.  In this case, the attacker could gain control of administrative rights over the system. The patch is critical for Internet Explorer 9 on Windows and Servers.
Vulnerability in Microsoft Data Access Components Could Allow Remote Code Execution This is a critical update that resolves vulnerability in data access components for Windows. If unpatched, could allow attackers to remotely execute code when the user visits a specific page and giving the attacker user rights over the system. The patch is addressed to all Data Access Components in Microsoft Windows and Server.

Non-Critical Important Patches
• Vulnerability in Visual Basic for Applications could allow remote code execution
• Vulnerabilities in Windows Kernel-Mode Drivers could allow elevation of privilege
• Vulnerability in Windows Shell could allow remote code execution
• Vulnerability in TLS could allow information disclosure
• Vulnerabilities in SharePoint could allow elevation of privilege
• Vulnerability in Microsoft Office for Mac could allow elevation of privilege
More information about Patch Tuesday 2012 can be found here.

 

Flash Plugin
Yesterday, there were some rumblings on the Adobe forums regarding a Flash 11.3 update.  As suspected, Flash released- overnight  - the new Flash 11.3.300.265 – perhaps for both Active X and Plugin versions.


This update is not on Adobe’s Distribution Agreement site yet, and no security bulletin has been published, but this update is on the Get Adobe page.

The Chrome 20 patch provided a Chrome updater. This is beneficial because it works like other updaters (Java) where Chrome monitors which level you are at and puts the update in the user's hands.

 

For enterprise customers, using the silent command line to deploy the Chrome patch across their environment (using the chrome_installer.exe), this update produces an error. The work around is to download and install the GoogleChromeStandaloneEnterprise.msi. Per Google, this is the stable release.

http://thwack.solarwinds.com/community/application-and-server_tht/patchzoneBack in the day, patch management had to be done using large spreadsheets, linking each patch to its specific version number in order to track what needed to be patched.  Tracking the software versions to be patched with a spreadsheet can be a nightmare for administrators.  Microsoft improved upon this process with the introduction of Windows Server Update Services (WSUS) in 2007, which made patching Microsoft updates easier.


Today, with the introduction of virtualization, administrators are faced with a different kind of problem: how do you distribute and deploy patches to multiple machines across the network when many of those systems are all shut down?  It is a very time intensive task for admins to turn on off-line physical systems or dormant virtual machines.  
Luckily, there are several approaches to solve for this issue,


• Administrators can announce a particular maintenance time were they can ask users to leave their desktops powered on.
• WSUS has a feature of notifying the user through balloon notification about the available update.  In this case, the desktop must be part of the WSUS automatic patch deployment processes.
• Administrators can use a remote wake up feature using a third party application, so administrators can switch on or reboot user systems remotely.  For virtual machines that are off-line, you can manipulate the bits on the VHD (virtual hard disk) file or you can power the machine on, update the machine using WSUS and then power the machine back down.


Scheduling maintenance time can cause network downtime and may become infeasible in large enterprise as it is difficult to determine whether the systems are turned off or on. And if your organization uses WSUS, end users can deny the update from happening by cancelling it.


Which is the best option?

Wake-On LAN might be the best alternative because you are able to reboot computers after the patch is applied.  Once more, if you are using a patch management tool, you can run a report to see if the patch was successful before you power-down the system.


How do you patch off-line servers & desktops? Please comment and share your experiences!

 

Sign up for alerts on PatchZone news & tips here.

The patching landscape: not a pretty picture.
The landscape of patching is messy.  We see 20 to 30 patches a month for Microsoft.  Those are predictable because they do come out on Patch Tuesday. There are a whole slew of other patches that come out in a month for Java, Adobe, Mozilla, etc. – the list goes on and on.  There are also many dependencies with 3rd party application patches.  For example, Flash is embedded in Adobe Reader, Chrome, and now in IE 10 later this year.  When there is a flaw in Flash, this causes these vendors to also issue patches, so we see a spiral of patch updates.

 

3rd party application patches are more difficult, because they are not predictable, so the workload of the sysadmin dealing with these patches is not predictable.  Almost always, these 3rd party patches are related to a security issue; bug fixes and feature enhancements are rolled out as planned releases.

 

Microsoft, on the other hand, does provide a fair amount of non-security related bug fixes.  About 1/3 of Microsoft patches are related to bug fixes – critical flaws in the product that cause performance issues or configuration settings (day light savings settings for example).

 

When should security patches be installed?
As soon as possible, but the sysadmin needs to take into consideration business need and risk of not updating the patch.

 

Internet facing patches need to be the top priority – a browser, Flash, Java, etc.  Why?  The whole world knows there is a security vulnerability and there are plenty of folks out there ready to exploit those vulnerabilities.  The longer you delay getting the patch deployed, the greater risk on your infrastructure.

 

Servers, especially web servers, mail servers – are an attack target.  Do these immediately.  Desktops/notebooks are less of a direct target.  However, the end user might do something naively or innocently.  Classic example – identity theft, email phishing skemes.

 

When should bug fixes be installed?
It depends.  If a bug fix comes out and if you are if you are not experiencing that bug or using that feature of the software, don’t install the patch.  Don’t increase the risk of instability into your system by installing unessary software.

 

Classic example - a year ago, a fix came out for Remote Desktop.  90% of users were not using.  Do you install across your entire environment?  The better approach is to check to see who is using this component, and apply the patch to those machines.

SolarWinds Patch Manager packages and tests 3rd party applications like Adobe, Chrome, Mozilla, iTunes, etc.  Lawrence, based on our past experience, which of these 3rd party apps are most vulnerable?

 

Lawrence: Most of patches in 3rd party application landscape are security related, and most are related to the user browsing content on-line, in real time.  Security updates that focus on Flash and browsers should be at the top of the list for patching. 

Sadly the majority of infections we have heard about in recent years were for these known vulnerabilities, meaning that a patch was available for Flash, Chrome, etc., but the patch was not deployed to the environment.

The other kind of applications are those used in an off line mode  – like downloading a document or video using Adobe Reader or QuickTime.  These tend to be less vulnerable.  The risk in these situations is end user does something inadvertently.


Has the increased use of content from social sites (videos, files) changed the frequency of threats from these applications? 
Lawrence: Video, in terms of AVI and Windows media files, is not risky because it takes a lot of effort to embed viruses.  Flash is the most frequently used medium for viewing video on line today.  It is not so much the risk from the Flash files, but ways to exploit Flash code through other methodologies.

Security threats for social media are not about direct attacks on players or browsers, but more around the social engineering aspects.  For example, fraudulent posts, and bad links that link to websites designed to steal information.  It is the users who are not savvy on these sites and they unknowingly are attacked.  It’s like the early days of AOL.  Everyone thought AOL was the internet – users did not know what they were doing.  Same situation we have today with Facebook.

 

Do customers need to be concerned about Browser/OS vulnerabilities for their mobile device?
Lawrence: Mobile software, iOS, Windows, etc. are as vulnerable as desktop software.  I have not seen incidences where mobile devices have been directly attacked.  We hear more about security protections on mobile data – password, encryption schemes.  Easy to use, consumer oriented security programs, like 4 digit codes to open your phone, are more of a threat and no amount of patching will solve for that.

Patching mobile systems is not so easy.  Hard to get updates to Windows phones with the mobile carriers being responsible for that – hard for Microsoft to get phones patched timely.  Android gets regularly updated for those people who are savvy enough to get the Android source and install it.  Most people don’t – it’s like installing a new version of Windows.

 

Will we see more patching for phone software?
Lawrence: Not likely until there are more cases of hacks.  I am surprised there have not been more attacks on iOS, because of the prevalence of iOS.  Is it harder to attack a mobile phone?  Maybe, maybe not.  Or perhaps it is the sophistication of the hackers – they don’t have the skill set yet to hack mobile.  It could also be that most mobile phones are protected via the carrier’s network – 3G/4G and not using wireless.

The patch management process varies from organization to organization based on infrastructure size, organizational policy and size of the organization. Smaller organizations may assign a single administrator to take care of patch management and the network, while large enterprise offices may have specific person or a dedicated team to deal with patch management. In all instances, the reason for patch management is to keep applications up-to date, mitigate vulnerabilities and to meet with compliance standards.

Patch management involves a series of responsibilities assigned to a person/team who handles patch management.  Responsibilities include:


• Identifying available patches or threats
• Deploying patches
Compliance assessment & reporting
• Educating stakeholders

 

Identifying the available patches is the first step in patch management.  The administrator should learn about their application landscape. They should be aware of software installed in the user desktops, servers etc. Only when the administrator has awareness about the environment, can they start identifying the necessary software that is needed to be patched.

 

Deploying patches involves testing the patch in test environment for possible errors and then deploying the patch.  After deployment it is necessary to monitor the application environment to determine whether the patch was successful – did it cause performance issues or was the patch deployed to all needed systems.

 

Compliance involves regular audit and assessment of whether applications have been patched for known security vulnerabilities  Audit reports help administrators understand:
• What systems need to be patched for a given vulnerability?
• Are the all systems in compliance?
One aspect of patch management is change management, which involves keeping in track of all updates.  So, if anything goes wrong there is back-out plan-documentation of the version and configurations to roll back to.  This documentation is also helpful in meeting audit requirements.

 

Education involves educating end-users as well management on why patch management is necessary.  With education, you can explain your logic for how frequently updates are made.  Updating regularly may cause system down-time which affects productivity, while not-patching increases the risk of vulnerability. So administrators need to educate users about patch management from security perceptive and also in-term of productivity.

System Center Configuration Manager (SCCM) Pros & Cons
SCCM is popular patch management software among most administrators. A single admin or group of administrators can use SCCM to control the deployment of software patches with ease.  One of the salient features behind the usage of SCCM by most administrators is the simplicity involved in the deployment of patches.
Below are some of the advantages of SCCM:
• The user can automate the software installation across all the systems in the environment
• SCCM directly pushes the updates to the systems without end-user needing to be involved or take action
• SCCM software updates can be scheduled
• SCCM can push updates to ultra-secure computers if proper credentials are provided

 

One may wonder if - SCCM can do so much why you will need a 3rd party Patch Management Software? Well, SCCM has its own disadvantages too. Below are some of the disadvantages of SCCMs:

• No immediate notification for failed updates.  For example, in the case of a user system being corrupted or in the case  of a previous installation failure the updates pushed may fail
• SCUP (System Center Updates Publisher) only provides updates for available catalogued patches.  Many application vendors like Adobe, Google, Oracle, do  not provide catalogs and in this case, the end user is forced to research when these patches become available, build the package, test it, and so on. There are quite a few manual steps to create a package with SCCM:
     -Research the patches needed.

     -Obtain the update installer, run the update installer, and document the pre-installation and post-installation conditions relevant to the update.
     -Create a package definition that encapsulates the metadata about the update, as well as the specific rules used to determine if the update is applicable, if the update is not yet installed, or if the update is already installed.

     -Publish this update package to the Software Update Point (SUP).
     -Synchronize the update package from the SUP back to the Site Server.
     -Build a Deployment Package (or add to an existing Deployment Package).

• Group policy edit and certification is not possible with SCCM
• Rollback of patches is not possible

 

Patch Management with a 3rd party Patch Management Tool
Most 3rd party patch management software seamlessly integrates with SCCM and adds more control and scalability in deploying patches and provides pre-built and tested updates for common 3rd party applications. Big vendors like Adobe, DELL etc. provide catalogs that can be consumed by SCUP. Administrators need to build update packages if the vendor doesn’t provide a catalog. Why spend time in testing, building, deploying (and potentially rolling back updates) when there is plenty of 3rd party patch management software vendors that can automate this task for you?

Importance of Patching Non-Microsoft Applications
Patch Management has become crucial part of IT administration and administrators fell the need to patch the third party software’s or risk the networks security.

Gartner states in its report - report - Top 10 Steps to Avoid Malware Infections that “IT organizations must strive for continuous improvement in vulnerability detection and rapid security patch management, especially in often overlooked non-Microsoft components that are Web-facing.” According to research released last year by the CSIS Security Group A/S, 85% of all virus infections occurred as a result of automated drive-by attacks created with commercial exploit kits – targeting 5 applications: QuickTime, IE, Adobe Acrobat & Reader and JRE.

All Internet-based applications, especially browsers and browser plug-ins (i.e., Adobe and Apple QuickTime), should be a top patching priority.  However, due to the un-predictable nature of 3rd party patches, deployment of non-Microsoft patches is often significantly slower and less organized.

 

How and when to patch Non-Microsoft products
Microsoft products can be patched with Windows System Update Service (WSUS).   However, non-Microsoft patches are not supported by WSUS and administrators can resort to manual patch process or purchase automation software for third party patch management. Manual patching can be done by pushing the software by manually preparing each server for patch deployment. Patch deployment to end-user can by deployed by using Group Policy along with Update Management. By editing group policy windows update agent can scan WSUS server for latest patches. Administrators can define when to deploy and scan the WSUS server for updates.

Another option is patching non-Microsoft products using System Center Configuration Manager.  This product provides the capability for creating the package, but does not provide the content itself – so administrators must still research, package and test the patches.

If going the manual route, administrators should plan patching based on business requirements (security/bug fix) and when they have time to schedule them.  To determine what needs to be patched, administrators can visit the third party sites that have the latest patch information.  Adobe, Oracle, and other ISVs normally describe the contents of the patch and sometimes flag the type of patch (severe, critical security update, bug fix, etc.).  Unless the patch is critical, the good option is to schedule the patch updates once in month so it reduces time spent on patching and prevents application performance issues related to patch updates.

 

Check out this table on Patchzone for the most recent patches available for 3rd party applications.

 

 

Here are some other links for reference:

Microsoft Security Bulletins  http://technet.microsoft.com/en-us/security/bulletin

Adobe Security Bulletins & Advisories: http://www.adobe.com/support/security/

Mozilla Security Bulletins http://www.mozilla.org/security/announce/

Apple Security Bulletins http://support.apple.com/kb/HT1222

Oracle Security Bulletins http://www.oracle.com/technetwork/topics/newtojava/overview/index.html

 

Automating patch management can save a lot of time
The manual route has its limitations.  Researching, building, testing and deploying patches are extremely time consuming tasks.  If your company has a large number of vulnerable servers, you should definitely consider purchasing patch management software.  What you should look for:


• Catalog of 3rd party patches –Most vendors support Adobe, Java and most browsers.
Before and after patch deployment scenarios – especially important for Java.
• A mechanism for scheduling where and when patches are deployed
• Support for creation of custom application packages.
Reporting for vulnerability assessment and patch compliance.
• Scalability – the software should scale to a large number of servers & desktops.