A disruptive day at the office could start when someone says: “Hey, I’m having problems with my OS; the last thing I saw is that there were a bunch of updates being installed”, and if we are not lucky and it is in fact an update that is generating the problem, we will need to get our hands dirty.

 

Getting in the situation when we know we need to rollback an update it is definitely not an easy situation. Maybe if we need to do it in our home computer it would not represent a big problem, but if we are facing a situation where we just approved the update for our entire organization, then things could get really messy.

 

In reviewing some of what we previously discussed in Patching Best Practices we will see that is almost a must to test our updates before deploying; but at this point, whether we tested or not, it does not matter, we need to solve this problem. However, you will see also that it is highly recommended to have ready a baseline image of our company user's system (maybe a virtual machine), this way we can rapidly start working with this image in order to solve the problem.

 

And of course, if we don’t have the baseline image, we can start working with the machine that appeared with the problem.

 

Start Working on the Problem

Having said the necessary about best practices, let’s start working on the problem. Here’s general guidance about what we need to do:

 

  1. If you are using WSUS, mark the recent approved updates as “Not Approved”. This will not remove the updates from computers where the update is already installed, but it will prevent them from being installed on any other systems.
  2. Use the baseline image to review the behavior when we have the updates approved.
  3. Perform some quick troubleshooting (Event Viewer, application logs) to understand a little bit more about the problem.
  4. Remove the updates installed in baseline image. We can use “Control Panel”; or “Windows Update” (in “Read More” section about “Installed Updates” sometimes the uninstall instructions appear).
  5. Verify that the problem is solved. 
  6. To extend this automatically for all users we can use WSUS or scripting:
    • For WSUS, configure the updates as “Approve for Removal” -- this will automate the process of uninstalling.
      • Even more, we set a “Deadline” to remove the update.
      • If we want to remove it as soon as possible, select a date in the past.
      • Important disclaimer: The “Approve for Removal” option is not available for several updates, so we might want to use the “scripting” option.
    • Scripting the update:
      • For Windows XP we can find the folder (hidden) for the update C:\Windows\$NtUninstallKB<number> and generate a script for uninstalling.
      • For Windows 7 we can use a script with “wusa /uninstall /kb:<number>

 

Handling Restore Points

“System Restore Points” are also a valid way to solve a faulty update deployment for Windows OS. A System Restore point is basically an operating system snapshot that is created whenever you make a significant change to your PC. One of the processes that can create restore points automatically is in fact Windows Update.

 

So, whenever we have a problem with a recent change, we can use a Restore Point to recover the exact state we have previous to our change.

 

Using System Restore Points

Using Restore Points does not have any big complication, just following a wizard; but the only trick is that we have to do it manually on every computer:

 

  1. Access “Computer” > “Properties”.
  2. Select “System Protection” and click on “System Restore”.
  3. This will open up a wizard, click “Next” in the first step.
  4. We will have the option for selecting the “Restore Point” to apply to our system, carefully select the proper one.
  5. Reboot the computer.

 

With that we’ll complete the necessary steps to solve our problem. But if we are going to depend on restore points, we must make sure we have the necessary configuration among all of our computers.

 

Creating Restore Points

To create restore points in our computers we can do it of course manually or by an automated process.

 

Unfortunately, the automated process is not all that simple. We will use Group Policy to distribute this configuration among our organization, but we will need a few command lines to accomplish this.

 

  • Creating Restore Points manually
    • Access “Computer” > “Properties”
    • In the left pane, click “System Protection”.  
    • Click the System Protection tab, and then click Create.
    • In the System Protection dialog box, type a description, and then click Create.
  • Creating Restore Points using Group Policy
    • Create a new Group Policy Object and link it to the Active Directory Container of your choice.
    • Create a new Task under [Computer Configuration\Preferences\Control Panel Settings\Scheduled Tasks]
    • Type a name and choose SYSTEM account to run this task.
    • In “Triggers” tab, click “New” and choose the period of time you would like to create a restore point.
    • In “Action” type “%windir%\system32\rundll32.exe" and in “Program/Script”: “/d srrstr.dll,ExecuteScheduledSPPCreation”
    • Click OK and complete the GPO creation.

 

For more information about “Restore Points” review the following link: “System Restore: frequently asked questions”.

WSUS Timeout Errors - When? and Why? - Eliminating and avoiding

 

This is the 5th and final part of this series on handling WSUS timeout issues, and now that we’ve removed unnecessary update approvals, addressed the issue of too many synchronized updates, and defragmented the filesystem hosting the WSUS database, we’re now ready to perform WSUS database maintenance.

 

In this article we’re going to talk about what the database maintenance activity does, how to get the database maintenance script, and how to successfully execute the script.

 

What Does WSUS Database Maintenance do?

The reference for the WSUS Database Maintenance activity is in the WSUS Operations Guide in the section titled Reindex the WSUS Database, and at its core, this is what the database maintenance script does. You could, of course, simply launch SQL Server Management Studio and reindex everything – but that would keep the database (and WSUS server) offline for a lot longer than is really necessary, potentially rebuilding indexes that don’t need to be rebuilt. The script evaluates the level of fragmentation and reindexes only those objects where the page density is low or the external fragmentation is high in relation to the index size. Specifically, the criteria are:

  • Page Count > 50 and Average Fragmentation > 15%, or
  • Page Count > 10 and Average Fragmentation > 80%, or
  • Average Page Space Used < 85% and the Average Page Space Used is low enough that pages can be freed up by rebuilding the index

 

How to Get the WSUS Database Maintenance script

The script can be obtained from the Microsoft Technet Script Center. The script is not an actual downloadable object, so you will need to copy and paste the script from the webpage into your local system.

 

CAUTION: There is a compatibility issue between the “Copy Code” link in the script center, and certain versions of something that results in upper-bit characters being inserted into the text stream rather than the simple spaces that are actually defined. The manifestation of this issue is discussed in this thread in the TechNet WSUS forum, and I was able to reproduce it today, using the “Copy Code” link with IE9 on Win7x64 and pasting into either Notepad or SQL Server Management Studio directly. When I copied the text directly from the display on the page, I did not encounter the syntax errors. I suggest you copy the text directly, rather than use the “Copy Code” link.

 

How to Use the WSUS Database Maintenance script

There are two ways you can invoke this script:

Either way, since you’ll need to download and install something if you’re using the WID, I suggest you go all the way and install the SQL Server Management Studio Express Edition. (You may find additional uses for this tool at another time.)

 

If you choose to use SQLCMD.EXE, and assuming you’ve saved the script as WSUSDBMaintenance.sql, the command to run the script from the command line is:

sqlcmd -S np:\\.\pipe\MSSQL$MICROSOFT##SSEE\sql\query –I -i <scriptLocation>\WsusDBMaintenance.sql

 

If you’re running this on Windows Server 2012 use:

sqlcmd -S np:\\.\pipe\MICROSOFT##WID\tsql\query –I -i <scriptLocation>\WsusDBMaintenance.sql

 

To run the command from SQL Server Management Studio, open the WSUSDBMaintenance.sql file, or paste the code directly into a New Query window connected to the SUSDB and click on “Execute” or press <F5>.

 

Be prepared! This script will take some time to execute. On my WSUS server (P4 2.8GHz hyperthreaded, 1.5GB RAM with separate spindles for OS and WSUS), the script took almost 8 minutes to scan the WID and identify 77 indexes that qualified for maintenance. To run the entire script took almost 19 minutes.

Today (Feb 7) Microsoft announced the forthcoming content for Patch Tuesday – Feb 11, 2013.

 

You can have Microsoft's security bulletins sent directly to you:

To receive automatic notifications whenever Microsoft Security Bulletins are issued, subscribe to Microsoft Technical Security Notifications.

 

Microsoft also hosts a webcast where they discuss the releases, typically the Wednesday after Patch Tuesday:

Microsoft will host a webcast to address customer questions on the security bulletins on February 13, 2013, at 11:00 AM Pacific Time (US & Canada). Register now for the February Security Bulletin Webcast. After this date, this webcast is available on-demand.

 

You can also follow the MSRC team at @MSFTSecResponse.

Number of Releases: 12

Number of CVEs addressed: 57

 

Critical Security Updates: 6 addressing vulnerabilities in Windows, Internet Explorer, and Exchange Server 2007 and 2010

Important Security Updates: 7 addressing vulnerabilities in Windows, .Office, Search Server 2010 for Sharepoint, and the .NET Framework

 

Updates are typically released by Microsoft at 10am PT (6pm UTC).

Configuring WSUS servers to synchronize relative to that time can be helpful in expediting availability of these security updates.

Windows clusters have always presented a special problem for those tasked with patch management. On one hand, clustering is very necessary. Clustering provides fault tolerance and is often the only way to protect a mission critical server or service. On the other hand, failover clustering compounds the difficulty of patch management.

 

Cluster Patching – The Old Way

Prior to the release of Windows Server 2012, if you wanted to patch a cluster (using only native tools) you had to move the clustered resources off of a cluster node, patch and reboot the cluster node, and then repeat the process for the other nodes in the cluster. The technique worked, but it required manual intervention and was tedious and time consuming. The manual nature of the process could be forgiven if patching was a one time process, but as any good admin knows, patching is an ongoing process.

 

Cluster Patching – The New Way

Those cluster administrators who dread Patch Tuesday will be happy to know that you don’t have to use a manual process to patch a Windows Server 2012 cluster. In fact, Microsoft has introduced a new feature called Cluster Aware Updating.

 

The tricky part of patching a Windows Server 2012 cluster is that cluster aware updating is not used by default. You have to enable cluster aware updating. Otherwise, the cluster will have to be patched using the same manual technique required by clusters running earlier versions of Windows Server.

 

How Does Cluster Aware Patching Work?

Before I explain how to enable cluster aware updating, you might be curious as to how the patching process works in a Windows Server 2012 cluster.  Cluster aware updating works similarly to the method used to manually patch a failover cluster, except that the process is automated.

 

Windows Server 2012 uses a round robin approach to updating cluster nodes. The process begins by identifying the cluster node that has the most free memory available. The clustered resources are moved from a random cluster node to the node that has the most free memory. After the clustered resources have been moved off of the cluster node, the node is placed into maintenance mode. At this point, the cluster node is patched, rebooted, and then taken out of maintenance mode. The process is then repeated for every remaining node in the cluster.

 

How Is Cluster Aware Patching Implemented

Cluster aware updating is based on the use of a new utility called the Cluster Aware Update Tool. The Cluster Aware Update Tool is automatically installed on all of the nodes in the cluster, but it is not active.

 

If you want to use the Cluster Aware Update Tool from outside of the cluster then you can install the tool by installing the Failover Clustering feature on any machine that is running Windows Server 2012 (you don’t actually have to create or join a cluster).

 

In order to perform cluster aware updating, the Cluster Aware Updating tool must run as a clustered server role. That way the update code can move from cluster node to cluster node as the updating process progresses.

 

How To Configure Cluster Aware Patching

To set up cluster aware updating, open the Server Manager and then choose the Cluster Aware Updating option from the Tools menu. When the Cluster Aware Updating tool opens, select your failover cluster from the Connect to a Failover Cluster drop down box, and then click Connect.

 

Once the connection to the cluster has been established, click on Configure Cluster Self Updating Options. This will cause Windows to launch the Configure Self Updating Options Wizard. Click Next to bypass the wizard’s Welcome screen and you should see a message telling you that the cluster isn’t configured with the Cluster Aware Updating cluster role.

 

You must now select the Add the CAU Clustered Role with Self Updating Mode Enabled to this Cluster check box. Click Next and you will be asked to set a self updating schedule. Microsoft recommends scheduling updates to occur at a time when there is the least possible demand on the
cluster nodes.

 

Click Next a couple more times and you will see a screen asking you if you wish to receive recommended updates the same way that you receive important updates. After making your decision, click Next and verify that the information displayed on the summary screen is correct. You can
complete the process by clicking Apply, followed by Close.

 

As you can see, it is relatively easy to implement cluster aware updating. After doing so, the process of keeping the cluster nodes up to date will be completely automated.

At first glance, one might look at the first ten Java 7 updates in eighteen months (through JRE7u11, u8 was not released) and feel like the sky is falling. But, let’s split those apart into functionality updates (planned) and security updates (unplanned), and maybe it's not quite as intense.

Functionality updates

First, the functionality updates, and what we see here are evenly spaced feature enhancements. These are the even numbered releases which signify a planned release.

Java 7 Functionality Updates.png

Security updates

Then the odd-numbered (and unplanned) security updates. Wow! Don’t those look just as regular as the feature releases!

Java 7 Security Updates.png

Let’s take a look at these six security update packages in a bit more detail.

 

The first shocker is that right out of the gate, in the first three months of existence, twenty security vulnerabilities were identified. What you might not know is that nineteen of those vulnerabilities also existed in Java 6, and had existed in Java 6 for the previous five years. Only one of these was unique to Java 7. Maybe even scarier: Nine of them still exist in Java 5! In fact, this is pretty much the story for almost all of these Java 7 vulnerabilities.


Here’s a chart showing the number of Java 7 vulnerabilities reported/fixed up to JRE7u11 that also existed in Java 6 and Java 5.

Java 7 vulnerabilities in older Java versions.png
Last Friday, Oracle released another security update for Java, (JRE7u13 and JRE6u39), and we can presume that JRE6u39 will be the last update for Java 6. If the above numbers shocked you, this security update might add a bit more of a shock. The update itself repaired 50 vulnerabilities, but eleven of them were only in the JavaFX module. The other 39 were in the Java Runtime Environment -- Yep! They fixed 39 more vulnerabilities! But only six of those vulnerabilities were exclusive to Java 7! Of the 39, 12 of them only existed in Java 6, and 21 of them existed in Java 6 and still exist in Java 5. One of the vulnerabilities didn't exist in Java 7, but only in Java 6 and earlier, so only 38 of them existed in Java 7.


So, if update the above table and add one more row and recalculate the totals, here's what we have today:

Java 7 u13 vulnerabilities in Java 6.png

Let’s put this in a very simple perspective:  over 80% of the reported/fixed vulnerabilities in Java 7 also existed in Java 6 (and have for the past six years) and almost half (49.6% to be exact) still exist in Java 5!

 

Which Java to keep?

What that means for patch administrators is that there’s no significant advantage to deferring the deployment of Java 7 and retaining Java 6, except in a case where an application has a known and verified dependency on Java 6. If that situation applies to you, be aware that after February, 2013, Oracle will not be publishing any additional security updates for Java 6, and you might want to have a heart-to-heart with that software vendor who still has Java 6 dependencies.

 

For some additional guidance on managing Java installations, see my Patchzone article.


Who’s responsible?

Excluding the 61 vulnerabilities that exist in Java 5, which Oracle definitely inherited from Sun, we might wonder about the other 38 vulnerabilities in Java 6. In fact, we can infer their “state of inheritance” by the version of Java 6 in which they were fixed: Every one of those Java 6 vulnerabilities existed in JRE6u26 and earlier. That is to say, all 80% of the Java 7 vulnerabilities that also existed in Java 6 were inherited from Sun!

 

Which, I guess, leaves us with this reality: Oracle inherited a problem in Java code security from Sun in 2009, and over the past eighteen months we’ve learned that a lot of them were not discovered, much less fixed, in Java 7. So we can’t really blame Oracle for creating them -- well a third of them are unique to Java 7 -- but there’s no doubt that Oracle is complicit in their perpetuation of a lot of bad code written in Java 6 and earlier.(Some of these vulnerabilities actually date back to Java 1.4 and earlier!)

So, I think I’ll temper my anger towards Oracle, at least as to their perceived ineptness in creating new code; however, as for the rest, I guess it really depends on where you are on the spectrum with regard to accountability for inherited conditions. Up until recently, I think, the prevailing wisdom in American popular thought regarding “inherited problems” would have placed all of this in the lap of Sun Microsystems.

On the other hand, regardless of where you point the finger, it’s definitely Oracle’s problem now.

WSUS Timeout Errors - When? and Why? - Eliminating and avoidingjavascript:;

 

We’re up to Part 4 now, and we’re going to look at defragmenting the database for WSUS. To recap in Part 2 we talked about removing unnecessary update approvals, and in Part 3 we talked about having too many synchronized updates in general. Update management is a necessary part of this process before optimizing the filesystem or database. In most of the situations I’ve seen and discussed with other WSUS administrators, it was the volume of data that was the primary culprit–-not the inefficiencies in the organization of that data within the database.

 

Defragmenting a disk drive can have a significant impact on disk performance; defragmenting a database file can have similar impacts on database performance. In the early days of Windows, defragmentation was something almost every Windows user, and certainly every system administrator, came to know and hate. The need hasn’t gone away, but Microsoft’s built-in defragmentation software did get better (actually, they licensed the software from one of those remaining vendors). Starting with Windows Vista, the defrag task is automatically created in the Task Scheduler to run at 1:00 a.m. every Wednesday morning. However, this task is disabled by default in Windows Server.  This is no great loss, though; a scheduled defrag task is totally useless for a database-based application anyway, because the database engine has a permanent file lock on the database files.

 

For a WSUS server using the Windows Internal Database, we need to take a slightly more direct approach. After completing all of the requisite update maintenance previously described in Parts 2 and 3 of this series:

  1. Stop the WSUS service. This is the “Update Services” service if you’re using the services MMC., or you can stop it from the command line using net stop wuauserv.
  2. Stop the database service.  Since we’re assuming a local database, that would be the Windows Internal Database service if you’re using the console, or net stop MSSQL$MICROSOFT##SSEE. (Note: If you’re using WSUS v6 on Windows Server 2012, the name of the WID service has changed to MSSQL$MICROSOFT##WID.)
  3. Defragment the drive containing the WSUS database. Navigate to Start -> All Programs -> Accessories -> System Tools and launch Disk Defragmenter. You can also launch from the command line using DEFRAG.EXE.
  4. Restart the database service.
  5. Restart the WSUS service.

 

It’s also worthy of note that the pretty graphical feedback that exists in Windows Server 2003, does not exist in Windows Server 2008 or Windows Server 2008 R2. In Windows Server 2008 R2, there is a progress indicator, but in Windows Server 2008 there is no active feedback at all. Of course, if you have an alternate disk defragmentation utility to use, that will work as well.

 

You can also script and schedule this task to run automated and unattended, and defragmenting a hundred gigabyte disk drive may take a long time–-even longer if it’s never been done. (This is another good reason to script and schedule it, so you’re disinclined to watch the progress state and get bored-–or depressed.)

 

By the way, if the database and content store are on different drives, it’s also worthwhile to defragment the volume with the content store too.

 

WSUS Timeout Errors - WSUS Database Maintenancejavascript:;

SolarWinds uses cookies on its websites to make your online experience easier and better. By using our website, you consent to our use of cookies. For more information on cookies, see our cookie policy.