Skip navigation

Having a “private cloud” deployed in an organization is far more common than it used to be a year or two years ago, when no one even knew what exactly this term meant. Or even more, we can say that you probably have already implemented a “private cloud” in your environment and you haven't noticed. The term “private cloud” is used to name a set of resources (hardware and software), that are delivered as a service over the network (if the network is the Internet, we call this a “public cloud”). The key aspect in this definition is the “as a service” section, since having these resources provided in this way we can guarantee agility, scalability, and reliability, among other benefits. To achieve and gain these benefits within our own “private clouds” (for example, our business application contained in two tiers, web and database servers), we must understand that several of our well-known processes for managing these services must change, including the patch management.

 

So, how do we apply system patches in a service that must be online almost “anywhere” and “anytime”? There are basically two approaches:

  1. Patching in-place.
  2. Re-building tiers with new updates.

Patching in-place

This is what we already know about applying system patches, updating the systems in the machines used to provide the service. But this doesn’t mean that we shouldn’t test the updates we are about to release; by definition itself of a “private cloud”, these services must have a proper stage for validating the maintenance process. To get a few best practices in this scenario, please refer to  a previous article of mine: “Patching Best Practices: Patch Scheduling

Re-building tiers with new updates

Also the definition of a “private cloud” tries to force us to de-couple the services/platforms provided from hardware and even operating systems. This definition will also demand to have some kind of “portability” in our systems, requiring for example to have automated and fast ways to deploy our entire service into a new set of machines. This process for “re-building tiers” will represent the way from which, by using an automated workflow ideally, we can replace the “out-of-date” OS used by the service with the “updated” OS without requiring any downtime. Automation is another key term for this approach. If we are planning to quickly replace operating systems, we need to have processes and tools that can give us these features and maintain availability of our systems. Microsoft provides a set of tools and platforms we can evaluate to accomplish this, starting with System Center Virtual Machine Manager 2012 (SCVMM).

 

SCVMM 2012 and Service Templates

SCVMM 2012 includes the use of “Service Templates” as the ability to group a set of Virtual Machines that are configured with several components, including applications, and that can be treated as one. The use of Services can have several scenarios where we can gain efficiency if we integrate it with SCVMM, one example is that we can scale-up our services dynamically whenever it is needed: we can add/remove virtual machines to support the load necessary for our business application without requiring re-defining the architecture or causing downtimes. 

Generating the Workflow with System Center suite

Besides the “Service Template” concept in SCVMM, Microsoft is seeking to provide an entire solution for this scenario of re-building the tiers with new updates. Microsoft calls it "Automated Fabric Patching for the Private Cloud". To achieve a full automation of this process, it actually requires the use of several MS technologies:

  • Hyper-V (virtualization platform)
  • System Center Operations Manager - for monitoring the environment
  • System Center Service Manager - for generating the process for updating the "private cloud" (named "fabric" in this case)
  • System Center Orchestrator - for connecting the dots and generating the automation workflow

 

Getting in the detailed description of this workflow, the “Automated Fabric Patching” works this way:

  • Add updates to baseline
  • Baseline add to host cluster
  • Run a compliance scan
  • Start remediation
    • Start maintenance mode
      • Move from HA VMs using live migration
      • Break mode in failover cluster
    • Install Windows updates
    • Restart the computer
    • Run a compliance scan
    • Stop maintenance mode

To get more information about this, take a look at this article: “Automated Fabric Patching for the Private Cloud”.

The Missing Approach: Virtualizing Server Applications

I have a big expertise in virtualizing applications and specifically using Microsoft App-V. The latest version of SCVMM includes the platform for “Server App-V”, which is the technology use to “encapsulate” server applications (like a MySQL engine) into one “bubble” and convert it to a portable service. Having this portability we can easily move around different OS without requiring downtime.

So far, Server App-V does not have full support with several server applications, but Microsoft is expecting to include more and more suitable platforms to be virtualized. For more information, please visit this article of mine: “A Quick Glance to Server App-V and Sequencing Server Applications

Update package definitions for WSUS can be built from a very rich set of rules -- 24 different rule types in fact -- but as a matter of practicality, almost every package you will ever build can be defined from one of only five rules. In this article we're going to discuss those five rules, where they should be used, and why. The five rules you need to know are:

  • Processor Architecture
  • Windows Version
  • Windows Language
  • Registry Key Exists
  • File Version

 

The first four of these rules are used, almost exclusively, in the Prerequisite Ruleset. The File Version rule is used in the Applicability Ruleset and the Installed Ruleset. (For background on the function of these three rulesets, see the previous articles How the Windows Update Agent Determines the Status of an Update and How to Troubleshoot Update Misbehavior Caused by Defective Rules.)

 

Processor Architecture

Just about every network in existence today has only two processor architectures in use: x86 and x64. But those are not the only processor architectures in existence. There is still a version of Windows Server 2008 for the Itanium; theoretically (but not yet in reality), the new Windows RT tablets on ARM, could, someday, become WSUS clients (not likely, but could!), and taking a page from the Year2000 lessons of yesteryear... x86 and x64 are not likely to be the only server platforms forever.

 

Any given update installer likely comes in one of three flavors:

  • Architecture dependent, either x86 or x64, with separate installers for each. Most software vendors have moved away from this technique, but some do still exist. It is critical that your package properly identify the processor architecture that the installer can be used with.
  • Architecture independent, with qualifications. The installer self-detects the x86 or x64 environment, but other processor-dependent requirements may exist. Most commonly this occurs on Windows XP systems, where the x86 installer requires Windows XP Service Pack 3, but the x64 installer only requires Windows XP Service Pack 2 (because there isn't a SP3 for x64).
  • Architecture independent, the installer self-detects the x86 or x64 environment and installs the proper files in the proper locations.

 

Windows Version

Most commonly the Windows Version rule is used to define the minimum operating system requirement for the installation, but it can also be used to exclude certain operating systems (e.g. a Flash Active X upgrade will never be applicable on Windows 8), and can also be used to restrict applicability to only workstation operating systems (WinXP/Vista/Win7/Win8) or only server operating systems (Win2003/2008/2008R2/2012).

 

Windows Language

Many installation packages, particularly updates, are language-agnostic, so it may seem that defining a language rule is unnecessary, particularly if you only have one language in your environment. But don't be too confident, as we're all but one acquisition, or one act of the legislature, away from having to support a second language overnight. Another lesson from the Year2000 collection -- if you only have English-language systems, and you're only considering your package in the space of English-language systems, then make sure your package is only detected by English-language systems until you're ready to test it on systems using other languages.

 

Registry Key Exists

This rule is particularly useful for qualifying updates as not applicable to any given system. A fundamental requirement of an update package is that the product is actually installed on a given system. Quite often a simple and reliable check for the absence of a product is to evaluate whether the correct vendor and/or product registry key exists in HKLM\Software. If the key is not present, the software is not installed (or it's a corrupted installation, in which case you still don't want to install the update).

 

File Version

This is the rule that does almost all of the heavy lifting in an update package. Since an update package is, generally speaking, a collection of one or more files to be replaced on a target system, the basis for whether any of those files needs to be replaced is the File Version attribute of each individual file.

 

In the Applicability Ruleset, we are interested in determining whether the file that currently exists on the target system is older than the file in the update package. If the current file is older, then it needs to be replaced; if not, then it does not need to be replaced. Therefore, in the Applicability Rules, we will always define a File Version rule with a LESS THAN operator, and compare it to the version of the file in the update package. You may need to define more than one File Version rules, if you are updating more than one file in the package.

 

In the Installed Ruleset, we are interested in determining only whether the file in the update package is already on the target machine. If the file on the machine is the same as the file in the package, then the package is not needed. As such, in the Installed Ruleset we will use the File Version rule with the EQUALS operator. (You can also use the File Exists rule and specify the file version attribute -- but who needs a 6th rule to learn, when 5 rules does just as good, eh? )

 

If you're interested in exploring the ability to create your own update packages for WSUS, download the 30-day trial of SolarWinds Patch Manager.

Building off the concepts learned in my previous post on rulesets, we can now use this information to diagnose unexpected behaviors in custom packages, or pre-packaged vendor packages. In this article we'll look at the two most common misbehaviors: An update that fails to detect as Installed when it should, and an update that fails to detect as Not Installed when it should.

Update not reported as Installed when expected to

There are a couple of scenarios in which an update can fail to report as Installed, one of which is related to the update package, the other related to the target system of the update, and it's important to make this distinction before trying to diagnose a perceived package defect.

Failing to install

The first scenario is when an update has been properly detected as Not Installed, but fails to successfully install. This behavior will manifest on the WSUS server as an update that fails to report as Installed. It will also manifest as an update that reports as Failed, but I have seen many scenarios in which this is exactly what was happening, but it was not recognized as an installation failure. An update that fails to install will continue to attempt to install at every scheduled installation event until either the cause of the installation failure is remediated, or the update becomes Not Applicable.

Failing to report as Installed

So an update that successfully installs, but fails to report as Installed is our focus in this post. When this behavior occurs, it is because the Installed ruleset has not been properly specified. One or more of the rules in the Installed ruleset is returning a value of FALSE, resulting in the ruleset returning a value of isInstalled=FALSE. The key is to determine which rule is returning false. There are two ways you can approach this, and using both ways is the best solution.

 

First, visually verify the truth of each rule in the ruleset. "Impersonate" the Windows Update Agent with Windows Explorer and Registry Editor and check the file and registry resources defined in the rules to see that they do exist, that the file versions are exactly a match, and that the registry values defined are identical.

 

Second, build and publish Metadata Only test versions of the package for each of the rules in the ruleset, and let the Windows Update Agent evaluate the packages. Do all packages detect as Installed? At least one should not, and that will identify the errant rule in the ruleset. Fix the ruleset in the original package and republish the package.

Failing to report as Not Installed

This is an update that reports as Not Applicable when it should be detected as needed. The techniques for diagnosing the problem are identical to the techniques just described in the previous paragraphs, except now you have two rulesets to troubleshoot -- the Prerequisite Ruleset and the Applicability Ruleset. However, we can take advantage of the fact that the PreRequisite Ruleset is actually optional (the Applicability Ruleset is not). So after visually verifying each and every rule (and presuming you've not already identified the errant rule), create test packages as before, except populate each rule in to the Applicability Ruleset only. At least one package will detect as Not Applicable, and that will identify the errant rule. Fix the appropriate ruleset in the original package and republish the package.

 

Understanding these behaviors can be very helpful when building your own custom packages, which I discuss in the next post.

On paper patching can seem quite straight forward.  You have a system running that needs a patch so you just need to install the patch and job done.  Unfortunately in the Enterprise environment it’s never that simple and there are a lot of obstacles that will be put in your path.

 

As I mentioned in my previous blog there is never a "one solution fits all" for Enterprises and it’s important that you understand your own environment and tailor your patch management to meet your own specific setup.

 

In my experience there are a few key areas that I believe are important to any Enterprise patch process.

 

1) Having a complete Asset Inventory

Before you can start patching you really need to know your Enterprise setup and what systems you have.  The mix of Operating Systems you are covering and what applications you are responsible for keeping up to date is important.  You may have common applications, for example Adobe Flash Player, which you have installed on all your machines.  There are many patching and software delivery tools that will collect an inventory as well.  You can then use this inventory data to build groups of machines with an affected application or component so you can see how many affected machines you need to patch.

 

2) Ensure that your patch has been through Testing

Any patch that you look to roll out will need testing as the last thing you want is to create an issue when you roll out a patch.  Not testing patches can have a negative effect on your patch management process so it’s important that you have a good range of systems that you can test on, either in a lab or nominated users in your enterprise who are happy to test patches, and if something goes wrong are able to remove or re-install them.

 

3) Plan how the Deployment of the patch is going to be done

Once you are ready to deploy you need to decide how aggressive to be with the deployment.  If it’s a critical patch you may want to target all machines within a few days whereas a less critical patch may be split into batches and deployed over the period of a week. In my experience a pilot of between 200-500 machines before the main deployment always helps to build confidence with your Service Managers that a patch is deploying successfully.

 

4) Dealing with any Exceptions

There are always going to be one or more machines in your Enterprise that are either too critical to patch, have issues with a particular patch or who have a specific maintenance window.  In these cases you need to make sure that you have these exceptions not only documented but also reviewed on an on-going basis. It's quite easy in a large Enterprise to lose sight of your exceptions if they are not documented and these may end up posing the biggest risk to your enterprise if not reviewed on a regular basis.

 

5) Produce reports that show progress

Now that the patch has been deployed the process shouldn't stop at that.  Have you checked that the deployment is actually working as it should and that the patch you are deploying is being applied successfully?  Many Enterprises will ask for progress reports of any patch deployment so it’s important that you produce these to show not only the progress of the deployment but also how many machines have actually been patched.  This builds confidence that you have successfully patched the system and the risk reduced.

 

That's just a few areas that you need to consider when looking at best practices for your patch process and you will need to make tweaks and additions to meet your own specific setup within your enterprise, I never said it was easy.

UPDATED LINK!!! Please follow us to this link and bookmark it for updates to the 3rd party catalog!

Table of third party patches - updated 06/21/2016

 

 

Simplify and automate these patches and more with SolarWinds Patch Manager.


Check out the attached spreadsheet for recent patches.

The Patch Manager catalog contains the following products:

 

AdobeAcrobat v9 (v10 and newer are provided by Adobe)Air v17-v13, v4-v2Flash v17-v10Reader v11-v8Shockwave v12, v11
AppleiTunes v12-v9QuickTime Player v7
CitrixReceiver for Windows Enterprise v3
CorelWinZip v19-v12
DameWareMini Remote Control v11-v9
FoxitEnterprise Reader v7, v6Reader v6, v5
GoogleChrome v42-v25Earth v7
MalwarebytesMalwarebytes Pro v2, v1
MicrosoftSkype v7, v6
MozillaFirefox v37-v3SeaMonkey v2Thunderbird v31, v24 and v17-v7
Notepad++Notepad++ v6
OperaOpera v29, v28, v12, v11
OracleJava Runtime v8-v6
RealNetworksRealPlayer Basic v16, v15
RealVNCVNC Server v5
UltraVNCVNC Server v1
VLCMedia Player v2
WinRARWinRAR v5
YahooMessenger v11


Change History (current month + 1):

 

5/4

  • Foxit Enterprise Reader 7.1.5.0425 5/4 (4/28)

4/29

  • Google Chrome 42.0.2311.135   4/29 (4/28)
  • Opera 29.0.1795.47 4/29 (4/28)

4/23

  • Malwarebytes 2.1.6.1022 4/23 (4/21)
  • Skype 7.4.85.102 4/23 (4/22)

4/22

  • Shockwave Player 12.1.8.158 4/22 (4/21)

4/21

  • Mozilla Firefox 37.0.2 4/21 (4/20)

4/20

  • VLC Media Player 2.2.1 4/20 (4/16)

4/16

  • Adobe  Air 17.0.0.144 4/16 (4/14)
  • Google Chrome 42.0.2311.90   4/16 (4/14)
  • Notepad++ 6.7.6   4/16  (4/15)
  • Notepad++ 6.7.7  4/16  (4/16)

4/15

  • Adobe Flash Player 17.0.0.169 4/15 (4/14)
  • Adobe Flash Player 13.0.0.281 ESR  4/15 (4/14)
  • Java Runtime Environment 8u45 4/15 (4/14)
  • Java Runtime Environment 7u79 4/15 (4/14)
  • Java Runtime Environment 7u80 4/15 (4/14)

4/9

  • Opera 28.0.1750.51 4/9 (4/8)

4/8

  • Citrix Receiver for Windows Enterprise 3.4.400 4/8 (4/1)

4/6

  • Mozilla Firefox 37.0 .1 4/6 (4/3)
  • Mozilla Thunderbird 31.6.0 4/6 (3/31)

4/2

  • Google Chrome 41.0.2272.118 4/2 (4/1)
  • Skype 7.3.60.101 4/2 (4/1)

4/1

  • Mozilla Firefox 37.0 4/1 (3/31)
  • Mozilla Firefox ESR 31.6.0 4/1 (3/31)

3/24

  • Foxit Enterprise Reader 7.1.3.0320 3/24 (3/23)
  • SeaMonkey 2.33.1 3/24 (3/23)

3/23

  • Mozilla Firefox 36.0.4 3/23 (3/21)
  • Mozilla Firefox ESR 31.5.3 3/23 (3/21)
  • Mozilla Firefox 36.0.3 3/23 (3/20)
  • Mozilla Firefox ESR 31.5.2 3/23 (3/20)

3/20

  • Google Chrome 41.0.2272.101 3/20 (3/19)
  • Malwarebytes 2.1.4.1018  3/20 (3/19)

3/19

  • Opera 28.0.1750.48 3/19 (3/18)

3/13

  • Adobe  Air 17.0.0.124 3/13 (3/12)
  • Adobe Flash Player 17.0.0.134 3/13 (3/12)
  • Adobe Flash Player 13.0.0.277 ESR 3/13 (3/12)
  • Adobe Flash Player 16.0.0.305 zip 3/13 (3/12)

3/12

  • SeaMonkey 2.33 3/12 (3/10)

3/11

  • Google Chrome 41.0.2272.89  3/11 (3/10)
  • Notepad++ 6.7.5 3/11 (3/10)
  • Opera 28.0.1750.40 3/11 (3/10)
  • Foxit Enterprise Reader 7.1.0.0306 3/11 (3/9)

How the Windows Update Agent (WUAgent) determines the status of an update

This article describes the basic logic of how an update distributed via WSUS (or AU, WU, and MU) is evaluated by the Windows Update Agent to determine the update's status on a given computer. This knowledge can be helpful for two reasons:

  • First, it can help diagnose why updates are misbehaving -- in the next post in this series we will describe exactly how to do this.
  • Second, understanding this logic process is critical to properly developing your own package content -- which we will also discuss in the third post.

For now, let's focus on the basics.

 

Two State Values; Three Valid Value Combinations

At the root of the entire logic stream are two state values named isInstallable and isInstalled. It is the logical combination of these two state values that determines how an update is reported to the WSUS server.

  • if isInstalled is TRUE, then the update is reported as Installed. (The value of isInstallable is not relevant in this case, but it should evaluate to FALSE.)
  • if isInstalled is FALSE and isInstallable is TRUE, then the update is reported as Not Installed. The state Not Installed actually consists of three sub-states, known as InstalledPendingReboot, Downloaded, and NotInstalled. In the WSUS native console, these states are all labelled as "Needed".
  • if isInstalled iS FALSE and isInstallable is FALSE, then the update is reported as Not Applicable.

The state of isInstalled=TRUE and isInstallable=TRUE is a logically inconsistent state, and should not occur in a properly constructed update definition.


Three Rulesets

There are three sets of logic rules that are evaluated by the Windows Update Agent in the process of executing a detection event. They are the Prerequisite Ruleset, Applicability Ruleset, and the Installed Ruleset. There are some very complex discussions in the WSUS SDK concerning the differences between the PreRequisite Ruleset and Applicability Ruleset, but both are used to evaluate the value isInstallable. The Installed Ruleset is used to evaluate isInstalled.

 

Prerequisite Ruleset

Primarily this ruleset is used to identify criteria that is absolute. For example, Adobe Reader X requires Internet Explorer 7 to be installed. If IE7 is not installed, then Reader X isn't an option and nothing else matters. Generally the Prerequisite Ruleset is used to identify those things that are absolute: Processor Architecture, Windows Version, Windows Language. It allows those sort of obvious things to be filtered out at the start. Updates for Windows 7 are ignored by Windows XP systems; updates for non-English systems are ignored by English-language systems; updates for 32-bit systems are ignored by 64-bit systems. If the logical aggregation of all of the rules fails, isInstallable is FALSE. Logically speaking in this condition, isInstalled could never be TRUE, so it is also presumed to be FALSE. The update is thus reported as "Not Applicable".

 

Applicability Ruleset

This ruleset is used to identify criteria that might or might not actually exist, which determines whether an update can be installed. For example, a product update for v1.1 requires that an earlier version of the product is installed; if it is not installed then the update is not relevant. More specifically, these rules are typically built to test individual file versions, so an update that replaces a file with version 1.2.3.4 requires that something prior to v1.2.3.4 is currently installed. If the earlier file is present, then it can be updated; if the product is not installed, and the file is not present, then the product cannot be updated. An update package may have only one rule (checking a single file version, for example), or it may have several rules. If the logical aggregation of all of the rules fails, isInstallable is FALSE; if the logical aggregation of all of the rules pass, then isInstallable is TRUE.

 

Installed Ruleset

This ruleset is used to determine if the update package is actually installed. It's not relevant to the Windows Update Agent how the update was installed, merely whether the identified criteria is true or not. It's important to understand that the WUAgent reports an update as installed, not because of an event that may, or may not, have actually occurred, but because of the state of the machine regarding that update. Typically a rule in the Installed Ruleset tests for the presence of a file with a specific version (e.g. is the file version EQUAL TO v1.2.3.4), but might also test for a registry key or value that uniquely identifies that particular update. For complex updates, there may be multiple tests for different files and/or registry objects. If the logical aggregation of all of the rules fails, isInstalled is FALSE; if the logical aggregation of all of the rules pass, then isInstalled is TRUE.


Evaluation of Rulesets

Returning back to our two state values, then:

  • If the PreRequisite Ruleset returns FALSE, then the update is NotApplicable.
  • If the Installed Ruleset returns TRUE, then the update is Installed.
  • If the PreRequisite Ruleset return TRUE, and the Installed Ruleset returns FALSE, then the Applicability Ruleset determines the status:
    • Applicability Ruleset is FALSE, the update is NotApplicable
    • Applicability Ruleset is TRUE, the update is NotInstalled

 

In the next post we'll look at how we can use this knowledge to troubleshoot an update package that is not behaving as desired or expected.

When Patching Can Do More Harm than Good

 

Conventional wisdom has long stated that patches should be tested before they are deployed. Although some organizations take the concept of patch testing to the extreme, many organizations do not do patch testing at all.

 

It is difficult to fault smaller organizations for neglecting to test patches. After all, IT departments are usually budget strapped, and busy administrators often lack the time and the resources to do comprehensive patch testing. Even so, blowing off patch testing can have consequences. There are a number of situations in which applying an untested patch can actually harm your systems.

 

Bug Introduction

The most widely accepted reason for patch testing is the simple fact that software companies occasionally release buggy patches. Sometimes the bugs within such a patch can break a mission-critical application. For example, back in 2010 Microsoft released a patch (MS10-024) for Windows Server 2008 and 2008 R2. This particular patch broke the Windows SMTP service, which caused certain types of servers to stop sending email messages. This is just one example of a major software vendor publishing a critically flawed patch. This incident is far from isolated. Although it isn’t every day that a major software company releases a patch that breaks a critical service, it does happen often enough to warrant patch testing.

 

Feature Removal

A much less talked about reason why it is important to test your patches is because there have been incidents of patches intentionally disabling or removing application features. Although this scenario might sound implausible, it actually happens on a fairly regular basis. For example, when Microsoft released Service Pack 1 for Exchange Server 2010, there were at least five features that were discontinued (http://technet.microsoft.com/en-us/library/aa998911.aspx#DiscRTMtoSP1).

 

Microsoft isn’t the only company that sometimes disables or removes features through patches. Application vendors have been known to do the same thing. To give you a more concrete example, consider an app that I run on my Windows smart phone.

 

The nature of what I do has me on the road on at least a weekly basis. Traveling back and forth to the airport so many times makes it easy to forget where I parked the car. I sometimes catch myself wondering if I am parked in Section A Row 2, or if that was where I parked last week. To make it easier to keep track of my car, I purchased a parking app that keeps track of your car's GPS location. The app worked great for several months until a new patch was released. The patch changed the entire nature of the app. Now, the app leads you to the parking lot, but it won’t show you where your car is located in the parking lot like it used to.

 

Crippled OS

Every once in a while a problem with a patch or with your patch management software might actually cripple an operating system. Over the last few years, I have seen at least a couple of situations in which someone applied a patch only to have the patch installation process crash halfway through, rendering the computer unusable. Although this isn’t a common problem, it does happen.

 

Bad Timing

Another good argument for doing at least a minimal amount of patch testing has to do with the patch deployment process itself. If an organization has a policy of testing patches before they are applied then it stands to reason that the Windows operating system will not be configured to automatically download and install patches at will. One reason why this is important is because the Windows operating system always seems to have impeccable timing when it comes to automated patch deployment.

 

I have lost count of the number of times that I have been out of town and needed to remotely connect to my computer at home only to find that Windows installed some patches and rebooted the system. Likewise, I have been in the middle of working on a project, gotten tired and gone to bed for the night, only to wake up the next morning and find that Windows was rebooted in the middle of the night as a result of the patch installation process.

 

Conclusion

As you can see, there are a lot of bad things that can happen if you allow patches to be automatically deployed without doing at least a minimal amount of patch testing. IT pros are busy, and we all know that patch testing takes time. But taking just a few minutes out to make sure that patches deploy correctly and don’t break anything in the process can save you a lot of trouble.