6 Replies Latest reply: Jun 17, 2012 5:06 PM by aalvarez RSS

What do you do when an Update Makes Things Worse?

aalvarez

Did you ever get a call from a user saying something like this? (Just after a released of several updates?) “I was just using the %COMPANYAPP% and is crashing all the time and I’m doing the exact same I was doing yesterday but today isn’t working”

 

Definitely is not an easy situation because most likely all users with the same baseline (OS, patch level and applications) will have the same situation. We can start troubleshooting but if this is a custom and internal application, how do we start? How can we remove the updates we just installed to all users? And how much time do we have if all the users are not able to work?

 

What I always try to do is the following:

  1. Have a baseline image, maybe in a virtual machine, to analyze and test the updates. This way we will also know if there’s a problem with just one user image.
  2. Take a quick troubleshooting (Event Viewer, application logs)
  3. Remove the updates installed. We can use “Control Panel”; or “Windows Update” (in “Read More” section about “Installed Updates” sometimes the uninstall instructions appears).
  4. Verify that the application is working again.
  5. To extend this automatically for all users:
    1. For Windows XP we can find the folder (hidden) for the update C:\Windows\$NtUninstallKB<number> and generate a script for uninstalling.
    2. For Windows 7 we can use a script with “wusa /uninstall /kb:<number>

 

Did you ever have to remove an update for your organization?

 
  • Re: What do you do when an Update Makes Things Worse?
    antwesor

    So far, there have been very, very rew instances where we had to remove a Microsoft Update. On those rare occassions that we do have to remove an update. Patch Manager has been the tool of choice and it seems to work very well for our organization of less than 1,000 computers. We also take the above troubleshooting steps before ruling out the possibility of a network configuration change or something other than the "patch" causing the problem.

     

    All in all WSUS does work well for small to medium organizations.

  • Re: What do you do when an Update Makes Things Worse?
    byrona

    We are a service provider and patch many of our customer systems as well as our own and there have only been a few times where we have had to actually go in and remove patches.  In the cases where we have had to remove patches we have done it manually by going into the Control Panel and removing them on the specific system.

     

    At this point a large percent of our infrastructure is virtual and we are looking at the possibility of doing VM snapshots on systems before patching them.  This will save us a lot of time if we do encounter problems as a result of patching.  We are in the process of rolling out a new enterprise patch deployment system and I have contacted the vendor with a feature request to have the product automatically trigger the VM snapshots pre-patching and ultimately be able to manage the snapshots for removal at later scheduled time as well.

    • Re: What do you do when an Update Makes Things Worse?
      aalvarez

      Hey byrona,

      Yes, using snapshots in VMs is vital for those cases. Also as an alternative to that for a physical machine would be the to run a system restore in Windows OS right?

      • Re: What do you do when an Update Makes Things Worse?
        byrona

        Yes, for physical systems that is what you would need.  Going forward patching solutions (much like many other enterprise solutions) need to broaden their perspective beyond just patching and look at the other pieces of that and provide automated mechanisms for managing snapshots or restore points as part of the process.

         

        All too often these products are just focused on the task that they do (i.e. patching) and fail to understand the needs of an IT and business environment (i.e. patching with easy fail-back options).

        • Re: What do you do when an Update Makes Things Worse?
          aalvarez

          Yes, I agree on that. There are tons of occasions where I see companies just looking for tools to solve their current problem and expecting that they won't find it again. Would be great if people don't focus only on "troubleshooting" but also in maintaining and operating they infrastructure pro-actively with the right tools and processes. 

  • Re: What do you do when an Update Makes Things Worse?
    stephen.black

    The context of this question causes me to wonder if anyone else is experiencing significant issues with SAM 5.0.1 since the most recent MS patching? We have a mess on our hands and are not sure if it was due to MS Patching or due to our upgrade from SAM 5.0 to SAM 5.0.1. Have been working with Support for a while but our VP of IT is not very happy at the moment.