WSUS Timeout Errors - When? and Why? - Eliminating and avoiding

WSUS Timeout Errors - Removing unneeded update approvals


In Part 3 of this series, we’re going to discuss the complications of having too many updates in your WSUS environment, and their contribution to causing timeout issues.


There are two situations in which we can find ourselves with too many updates. The first is a planning complication; the second a maintenance complication.


Planning Decisions

When a WSUS server is in the planning stages, one of the determinations that must be made is proper identification of the Product Categories and the Update Classifications that need to be selected for synchronization. Generally speaking, there are two rules you should follow when making these selections:

 

Select all Update Classifications except "Drivers"

Do NOT select the "Drivers" classification! (Yes, I repeated myself; it’s that important!) The “Drivers” classification contains over 30,000 device driver update packages, and for most organizations, none of those packages will apply to any installed hardware. If you have a server with the “Drivers” classification selected, you might consider rebuilding that server without it.

 

Select only the Product Categories for products that you actually have installed and need to deploy updates for

Do not select Product Categories you might have someday; do not select Product Categories that can no longer be patched. To determine if a product can no longer be patched, go to the Microsoft Support Lifecycle page, and look up a product. If the Extended Support date has passed, then there are no new updates for that product. Point the product to Microsoft Update, make sure all of the available patches are installed, and leave the Product Category unselected. You’re done patching that product … forever.


Maintenance Activities


Unselect the Product Categories for products that no longer exist in your organization, or have reached the end of the Extended Support period

Presumably, based on the guidance in the previous article, these updates have already been declined. The second part of that step is to update the synchronization rules. This won’t physically remove the updates from the database, but the change will hide them from the console and, more importantly, from many of the queries that look at all updates, including the declined updates. This will reduce the number of updates that need to be processed by client detections, downstream server synchronizations, and the Server Cleanup Wizard.


Regularly run the Server Cleanup Wizard

The Server Cleanup Wizard provides a number of services that can help reduce the total number of updates in the WSUS collection:

    • It deletes old revisions of updates.
    • It declines superseded updates that do not have active approvals.


Better Performance

The two most significant factors in the content of the WSUS database that affect performance are the total number of synchronized updates and the number of those updates that have approvals.


How many updates should you have?

It’s difficult to put a finite quantity on this value, because it does depend on the required selections for Product Category, Update Classification, and Update Languages. However, as a comparison point, on my English-only WSUS servers, synchronizing all Windows operating systems, most of the “BackOffice” server applications, two versions of Office, and Definition Updates for Defender and MSE, I have fewer than 6,000 updates on my server today.


How many approved updates should you have?

On any given day over the past several years, I have rarely seen the number of approved updates on any one of my WSUS servers exceed 10% of the total number of updates synchronized. Your mileage may vary, but if its more than 15%, you probably want to go back and look at

Part 2 again.

 

Deleting Updates

Just to entice you, you can physically delete other updates from the WSUS database. It can be done via API calls with code you write, or by native features of SolarWInds Patch Manager.

 

WSUS Timeout Errors - Defragment the filesystem

WSUS Timeout Errors - WSUS Database Maintenance

In Part 1, I introduced the three manifestations of timeout errors in a WSUS environment, and identified the four practices that will help eliminate them. In this article we’re going to talk specifically about the practice of removing unnecessary update approvals. This is a practice that all WSUS administrators should adopt, whether timeout errors are an issue, or not -- it’s just good practice.

 

Why do approvals matter?

Update approvals collect up over time, to the point where eventually they achieve critical mass and start interfering with efficient database operations within the WSUS database. WSUS was never intended to be a run-it-and-forget-it type of application, and it does require regular administration and maintenance.

 

How do approvals impact performance?

In WSUS, there are four entities of interest that are impacted by approvals: Updates, Groups, Computers, and Approvals. As the counts of each of those entities increase, the combinations increase exponentially. This causes longer processing times within the database when returning queries to the console or providing synchronization event data to a downstream server. Eventually the result doesn’t come back fast enough, and the console or downstream server gets tired of waiting.

 

One of the ways we can control this performance behavior  – and in an amazingly significant manner – is by managing the number of approvals that exist on the WSUS server. Generally speaking, a well-maintained WSUS server will rarely have more than a few hundred approved updates. So, as a starting point, if a WSUS server has more than 500 approved updates, it’s a candidate for a review of the approvals.

 

Updates that should have approvals removed

There are three classes of updates that we can target for update approval removals:

  • Products that no longer exist in the environment. If you’ve upgraded all of your Windows XP systems to Internet Explorer 8, and retired all of your Windows 2000 systems, then you really have no use for any updates for IE5, IE6, or IE7, or for Windows 2000. Decline them.
  • Any update that has been superseded by a Service Pack that has been fully deployed to your organization.  Updates superseded by a service pack that is installed everywhere have no useful value. Decline them.
  • Any other superseded update that is reported as 100% Installed/Not Applicable on the WSUS console. If superseded updates are 100% Installed/NA, then they will never be used (read: deployed) to any system ever again. Decline them.

 

This third group is usually the most difficult to identify. One way to do this is to:

  1. Filter the All Updates view with Approval=”Approved” and Status=”Installed/NotApplicable”.
  2. Enable the Supersedence column display
  3. Sort by the “Installed/Not Applicable Percentage” column. Alternatively, you can also sort by the Supersedence column. You may prefer one method over the other, and either will serve the purpose.
  4. Select the updates reported as 100% Installed/NotApplicable that are identified as superseded.

 

PatchZone - SupersededNAUpdates.png

Note: If you are using SolarWinds Patch Manager, you can build a permanent update view with these settings, and then do a Select All on the list of updates. I describe how to do this in a Geekspeak Blog Post.

 

Having identified the updates to be declined:

  • If you do not have downstream replica servers, you can Decline these updates directly.
  • If you have one or more downstream replica servers, do NOT perform a mass decline of hundreds of updates on your upstream server. This is also a known cause of downstream replica server synchronization timeout errors. Rather, in this case, you’ll need to spread this activity out over several sessions so as to not overload the number of approval events that need to be synchronized to the replica servers.

 

This procedure should then be performed on a regular basis. Monthly would be optimal, but you may find that removing approvals on a quarterly basis is sufficient to maintain performance of your WSUS server.

 

WSUS Timeout Errors - Too many updates

WSUS Timeout Errors - Defragment the filesystem

WSUS Timeout Errors - WSUS Database Maintenance

UPDATE: 3/1/2013

The fun never ends. Yet another zero-day vulnerability in the current patch-revisions of the Java Runtime Environment. Security research firm Fireeye reports that there are active exploits in the wild against both JRE7u15 and JRE6u41. The most significant point of this is that JRE6u41 isn't likely going to get fixed, as JRE6 expired support yesterday. Presumably JRE7u17 will be available shortly, but at a minimum you need to disable Java in IE or use a browser that prompts before launching a Java applet in the browser.

 

Note also, since the release of JRE7u10, you can now disable Java in Internet Explorer using the Java Control Panel for JRE7, but the UI is a bit flaky. You'll need to select the "Microsoft Internet Explorer" option, and then press spacebar to disable the functionality. And even then, this may not be enough, there is some discussion and controversy around whether this actually works or not. Seriously, if you're not actively using a Java-based application on your computer right now - today! - I would suggest removing Java completely. I have!

 

Disable plugin for JRE7 in IE.png

 

UPDATE: 1/31/2013

Previously I wrote in this article, incorrectly, that the vulnerability fixed in JRE7u11 was also present in JRE6. This is not correct. However, subsequent research has also uncovered the fact that 78% of the Java 7 security vulnerabilities that have been fixed since its release in July, 2011, are also present in Java 6. As such, for those that might have been inclined to stick with Java 6 to avoid the security issues with Java 7, that's not going to be of any real help. Also, security updates for Java 6 will no longer be published after February, 2013, so if you need Java, migrating to the latest release of Java 7 and removing Java 6 is definitely the best approach.



It’s been yet another busy 72 hours in the land of Java, although, by now, a lot of people have become quite accustomed to this rat race. The latest issue was reported Friday, by a number of sources, that an active exploit of an issue in the Java Runtime Environment (JRE) v7 update 10 was identified. Unlike past times, though, Oracle responded quite rapidly, and on Sunday released an update: JRE7v11 – which, unfortunately, doesn’t fix all of the identified vulnerabilities, but does fix the one being exploited. That update was published to the SolarWinds Patch Manager catalog yesterday.

 

However, be aware that the same vulnerability also exists in the Java Runtime Environment v6, and Oracle has not released a patch for that vulnerability, so don’t be surprised if an active exploit for JRE6 shows up in the next day or so.

 

In the meantime though, here’s an Action Plan for dealing with Java.

 

Best Solution: Uninstall Java

If you have machines that do not need the Java Runtime Environment, then uninstall it. Completely! If you have Java 7 and Java 6 installed (or any other older versions of Java) uninstall the older versions! Consider it one of those applications that only “special people” get to install. SolarWinds Patch Manager has tools that can you leverage to do mass uninstallations of Java across your entire organization.

 

Good Solution: Disable Java

Java is a pretty popular development environment and has been around a very long time. As a result, the reality is that almost every organization has at least one business critical application that requires the Java Runtime. Consider, though, disabling the Java Runtime on those systems that need it, and only explicitly enabling it when the application(s) that require it need to be used. This can be done through the Java Control Panel.

 

Minimum Solution: Disable Java Plugins in the Browsers

Sometimes, though, it’s just not practical to disable Java. Some business applications are used on a daily basis, or the process of enabling and disabling Java don’t work well for some users. At a minimum, though, you should disable Java in web browsers. This is fairly easy to do in almost every environment. Here are links for disabling Java in each of the major browsers:

 

Required Solution: Patch Java!

If you don’t uninstall Java, and regardless of whether you disable it completely, partially, or not at all, absolutely you need to keep it patched with the most current updates. Failing to apply this JRE7u11 patch can result in the installation of malicious software on PCs in order to steal identities or make infected computers part of a network used to attack websites.

 

To help all IT Pros with this challenge, today SolarWinds has updated the Patch Manager evaluation edition to include this latest JRE7 update. Previously only an older version of JRE6 was available for evaluations. We also know that the vulnerability exists in JRE6, and hopefully there is an update for JRE6 coming soon from Oracle, so when that update becomes available, we will also make it available to all evaluation users. Effective today, you can fully patch every JRE7 system in your network using a 30-day free trial of SolarWinds Patch Manager.

 

Oh, and, it takes a lot less time using Patch Manager! You can be done in as little as a few hours, typically overnight for most organizations – as opposed to days if you’re using scripts or doing it manually. Check out this video for a quick look at how Patch Manager makes patching Java so much easier.

 

If you need assistance, the Customer Portal is available for customers and evaluators can email direct to itmanagement@solarwinds.com. In addition, you can post to the Thwack Patch Manager forum, which I continuously monitor. Whether it’s completely uninstalling Java, or patching the installations you keep, Patch Manager has the ability to help you do both.

 

One last note, this new JRE7u11 update also has new protection mechanisms for dealing with unsigned Java applications, and will notify the user if one is encountered. This is a function of elevating the default security level from “Medium” to “High”, and is discussed further in the JRE7u11 Release Notes.

In several conversations during the past year I’ve observed an increasing number of occurrences of “Timeout Errors” with regard to Microsoft WSUS installations which seems to add complexity to WSUS patching.

 

When? and Why? Timeout Errors Occur

So typically in microsoft WSUS patch management, these timeout errors manifest in one of three ways:

 

Using the console to retrieve data for display

If a large number of updates is required to satisfy the query, or the complexities of the relationships between updates, approvals, target groups, and computer update status result in extra execution time, the console gets tired of waiting on the database.

 

Using the Server Cleanup Wizard

This issue manifests as a result of the “Delete Unneeded Updates” task, and typically is seen on servers that have been operational for a long period of time where the Server Cleanup Wizard has never been used.  It can also manifest on WSUS servers that just have too much data. Deleting data from a table in SQL Server is a fairly ‘expensive’ operation all to itself, and because there are constraints on whether updates can be deleted by this task, those constraints must be validated for each update identified as a candidate for deletion. The amount of execution effort required to validate these constraints can also be time consuming. Fundamentally this is a result of ‘too much work to do’.

 

Synchronizing from a replica server

Synchronizing updates from one WSUS server to another is a fairly simple process, but in the case of the replica server, it’s also necessary to synchronize approvals and declines. However, the WSUS server does not synchronize state data, but rather synchronizes events. Synchronizing an approval is a per-group event (number of updates approved x number of groups affected). Declining an update is the inverse of the approval (number of update approvals removed x number of groups affected PLUS the actual decline event). There are two very specific scenarios in which replica timeouts are being observed.

  • The first is caused because the upstream server simply has too many updates approved. Typically this scenario manifests when the replica server is first installed, as a result of the initial synchronization.
  • The second is caused because the upstream server has too many approval events to synchronize. Typically this scenario manifests when a large number of updates have been mass declined on the upstream server.

 

Eliminating and Avoiding Timeout Errors

As seen, the specific internal behaviors of the WSUS database that contribute to these scenarios are somewhat different in each case; but in every case, preventing these timeout errors can almost always be achieved by engaging in four maintenance “best practices” on your WSUS server.

  • Removing unnecessary approvals
  • Removing excessive updates
  • Defragmenting the filesystem hosting the WSUS database
  • Using the WSUS DB Maintenance script

 

Over the course of the next four weeks we’re going to dive into greater detail on patch management on each of these four highly recommended maintenance practices, and talk specifically about WSUS patch management and what each practice provides to the WSUS environment to avoid timeout errors, and how to ensure these tasks are performed correctly.

 

WSUS Timeout Errors - Removing unneeded update approvals

WSUS Timeout Errors - Too many updates

WSUS Timeout Errors - Defragment the filesystem

WSUS Timeout Errors - WSUS Database Maintenance

Adobe has published new catalogs for Reader v11 and Acrobat v11.

 

They can be found here:

 

Acrobat:  http://armmf.adobe.com/arm-manifests/win/SCUP/Acrobat11_Catalog.cab

Reader:  http://armmf.adobe.com/arm-manifests/win/SCUP/Reader11_Catalog.cab

Last week (Jan 3) Microsoft announced the forthcoming content for Patch Tuesday (today) – Jan 8, 2012. (I think it snuck up on all of us!)

 

You can have Microsoft's security bulletins sent directly to you:

To receive automatic notifications whenever Microsoft Security Bulletins are issued, subscribe to Microsoft Technical Security Notifications.

 

Microsoft also hosts a webcast where they discuss the releases, typically the Wednesday after Patch Tuesday:

Microsoft will host a webcast to address customer questions on the security bulletins on January 9, 2013, at 11:00 AM Pacific Time (US & Canada). Register now for the January Security Bulletin Webcast. After this date, this webcast is available on-demand.


You can also follow the MSRC team at @MSFTSecResponse.

Number of Releases: 7

Number of CVEs addressed: 12

 

Critical Security Updates: 2 addressing vulnerabilities in Windows, Office (2003 & 2007), Expression Web 2, SharePoint Server 2007, and Groove Server 2007.

Important Security Updates: 5 addressing vulnerabilities in Windows, .NET Framework, and Operations Manager 2007

 

Updates are typically released by Microsoft at 10am PT (6pm UTC).

Configuring WSUS servers to synchronize relative to that time can be helpful in expediting availability of these security updates.

What is Flame?

Defending network endpoints against malware tends to be a high priority in almost any organization. Although most of the major anti malware products do a good job, current malware protection is anything but perfect. Perhaps the best evidence of this is flame malware. Flame is an especially sophisticated piece of malware that has been used in targeted attacks in the middle east. There are a number of things that make flame unique, but the most alarming aspect of flame was that it has been confirmed that flame had infected high level systems for several years before it was detected. Flame was also unique in its size. Flame tipped the scales at a whopping 20 MB and consists of an estimated 750,000 lines of code. By way of comparison, most viruses have fewer than 150 lines of code. All of that code allows Flame to record audio, screen captures, keyboard activity, and even Skype conversations.

 

How does Flame work?

Of course these capabilities are not unique in and of themselves. There are legitimate parental control applications that utilize similar capabilities to allow parents to monitor their children’s computer usage. What was unique was the way that Flame managed to work its way into high level computer networks. Flame’s authors reportedly went to great lengths to disguise Flame as a legitimate Content Management System platform. Furthermore, the code was signed using a counterfeit Microsoft digital signature. Flame’s developers found a Microsoft Terminal Server Licensing certificate that had been accidentally authorized for code signing. Because this certificate used a relatively weak MD5 hash, the Flame developers were able to reverse engineer the certificate and use it to digitally sign their own code, giving the illusion that the code had come from Microsoft.

 

Risk of Flame impacting an organization

The odds of an organization becoming infected with Flame are slim. After all, Flame was designed for use against very specific targets. Even so, other malware authors now know that someone succeeded in creating a sophisticated form of malware that appeared to have come from a reputable source and that alluded detection for years. With that in mind, the big question is how organizations can protect themselves from similar, copycat malware that might be discovered in the future and that could potentially already exist.

 

How to protect against malware attacks

The best way to protect against such a sophisticated form of malware is to practice defense in depth. There is no such thing as a perfect security product. None of the antivirus products were able to detect Flame. The only way to guard against this sort of malware is to follow a very rigid set of security best practices.

 

Keeping your systems up to date with the latest patches is a good first step in preventing malware from being able to steal sensitive data. It is equally important however, to use a reputable patch management system. Remember, Flame posed as legitimate Microsoft code. It is therefore conceivable that a future exploit could pose as a Microsoft patch.  As such, it is critically important to use a patch management system that can be trusted to download patches directly from Microsoft. It is also a good idea to get into the habit of cross referencing patch numbers with TechNet to make sure that patches are legitimate prior to deploying them.

 

Another lesson learned from the Flame exploit is to be careful about where you download software from. Patches, drivers, and other updates should be downloaded directly from the vendor, not from a third party Web site. Flame posed as a legitimate CMS platform. While I don’t know where Flame was downloaded from, I am relatively confident that it was not downloaded directly from the Microsoft Web site. Microsoft and most other reputable software vendors go to great lengths to make sure that the code that they make available for download is not infected with malware. If you attempt to download a Microsoft patch from a non Microsoft Web site, there is a very good chance that you will end up downloading malicious code.

 

Although there is no method that is 100% guaranteed to keep malware off of your systems, being careful to download code from reliable sources and scrutinizing applications, drivers, and patches prior to installing them can go a long way toward keeping you safe from infection.

SolarWinds has now made our Diagnostic Tool for the WSUS Agent available for download with no registration requirement.

 

Just go get it and use it! :-)

SolarWinds uses cookies on its websites to make your online experience easier and better. By using our website, you consent to our use of cookies. For more information on cookies, see our cookie policy.