5 Replies Latest reply on Nov 7, 2014 8:57 AM by Lawrence Garvin

    Best Practices for "Catching up a Windows Environment" for patches.

    sinjen

      I have about 800 machines, a mix of Windows 7, and server versions from 2003 to 2012 R2. The ratio of workstation to server is about 50/50. I have been "trying" my best to get this environment caught up on patches for the last several weeks. Patching has been neglected over the past few years and the environment is kind of all over the place patch wise. Full disclosure, I'm familiar with WSUS in that I know how to release patches and pull reports and my knowledge of Solar Winds Patch Manager is about the same. I have yet to become experienced enough with Microsoft patching to workaround all the nuisances. For instance, yesterday, I spent an hour trying to figure out why a server was missing four patches. This didn't make sense since I had released all needed/failed patched that either did not have a superseding patch or it was the patch that superseded another. What was even more unnerving is that when I looked up the patches that were missing, they were superseded by newer patches (that I had already released). It turns out that Microsoft PULLED the newest version which is why it did not apply to the machine, causing it to report the older superseded patches were needed. Very frustrating.

       

      First question, to catch up an environment where patching has been spotty, should I just release ALL "needed/failed" patches including all the superseded patches that are reported by my machines? What advice would you give on methodically working through this environment to get everything caught up? We are a very small shop and I have many other projects to tend to (I'm sure that sounds familiar to most of you), so I'm looking for the most automated workflow that I can implement. For workstations, I can push updates pretty much weekly without much worry, for servers I would like to get to a regular monthly schedule which most of my servers can handle on an automated schedule. I'm interested in hearing all points of view from you Solar Winds Patch Manager/Wsus gurus on how you run your patch management solution/s.

       

      Do you use GPO's to let WSUS handle any updating?

      Do you exclusively use SPM and always schedule your installs?

      Do you approve all needed updates including superseded updates?

      What do you do about machines that aren't in your domain, but you still have to patch them?

      What about machines that need extra care such as Exchange, where you can't just let all CASHUB and Mailbox servers go off at once?

       

      I'm not really looking for a hugely detailed explanation, I've been in IT for several years, but patching has always been an after thought at most of my jobs and I was never directly responsible and when it did come up it was a huge ordeal and a TON of work (after hours usually) to get the environment patched. I am now and am interested in doing this the right way if such a way exists. So high level advice is what I'm after.

       

      Thanks for your time.

       

      -Bo

       

      Edit:

       

      Environmental Information:

      Most machines are in a domain and have Solar Windows WMI providers installed and working, but there are a couple machines where local admin credentials will need to be used.

      WSUS role and Solar Windows 2.0.2207.2 are running on the same 2012 R2 server.

      No additional WSUS instances are in the mix.

       

      Message was edited by: Bo S.

        • Re: Best Practices for "Catching up a Windows Environment" for patches.
          KMSigma


          I'll try to give some insight into your questions based on my years with a large corporation where I was both a Network and an Exchange Administrator and my work with Patch Manager since I've been with SolarWinds.  These are just a few of what I'd consider best practices with the limited understanding of your environment.  You'll probably have to tweak a few.  Others may offer their own "nuggets" of information, so I'd recommend listening to everyone and deciding what's best for your organization.

          Do you use GPO's to let WSUS handle any updating?

          Yes.  If you have Active Directory and you have a logical breakdown of your Organizational Units, then by all means use GPO's to handle pushing these settings.  Of course for those non-domain-joined systems, you'll have to do something slightly less different (I'll speak to that more later.)

          Do you exclusively use SPM and always schedule your installs?

          Not necessarily.  For workstations, I've always been a fan of just letting them patch on their own (on whatever schedule you decide) and holding back a VM or two of your standard image that you can use for testing prior to pushing patch approval out to everyone.  I'm going to bundle the server talk into the next question.

          What about machines that need extra care such as Exchange, where you can't just let all CAS/HUB and Mailbox servers go off at once?

          For Servers, I'm much more of a fan of the "Download, but let me choose when to install" option.  This is especially true for things like Exchange Servers which now compile some information during the installation process.  I see this as doing two things: 1) it cuts down on the time necessary to download patches and 2) it requires manual intervention on behalf of someone who can then properly test said system when the patching is complete.  Specifically regarding Exchange Servers, I'm a huge fan of this because some of your configuration changes (changes to the .config files or OWA pages for example) can be erradicated by some patches.  Having a knowledgable person on the hook for handling this always made sense to me.

          Do you approve all needed updates including superseded updates?

          This one can be a little "off" at times since Microsoft does occassionally remove specific patches for one reason or another.  I'm happier with making sure that there are Automatic Approval (WSUS) rules for Security, Critical, Update Roll-ups, and Definition Updates (your mileage may vary).  This way if I have something at the next sync that is needed, I've got it ready for my computers.

           

          Overall, I'm a personal fan of approving anything that is at the top of the "supersedence" path (either the most recent version of a patch or something that doesn't have a supersedence marker) and denying anything that's been superceded.  The new version of WSUS (2012 R2) has a nice cleanup feature that you can run with PowerShell to clean-up the updates that have already been superseded.

           

          Get-WSusServer -Name wsus.server.domain.local -UseSSL -Port 8531 | Invoke-WsusServerCleanup -DeclineExpiredUpdates -DeclineSupersededUpdates -CleanupObsoleteComputers -CleanupObsoleteUpdates -CleanupUnneededContentFiles

           

          I run the above daily (some time after my daily sync is scheduled) which also does some general housekeeping with regard to disk space.  This can also be done for earlier versions of WSUS via the COM Object, but I don't have that code handy at the moment (sorry).

           

          If you want to get everything ready for a catch-up process, you can approve everything, wait for it to download, and then clean up unnecessary updates.  This will cause significant bandwidth and disk consumption, but is an option.

          What do you do about machines that aren't in your domain, but you still have to patch them?

          I use the Windows Update Local Policy Management from within Patch Manager to configure them with settings that "match" those that I use for domain-joined machines.  That way everything reports back.  You can also "tweak" the policy so that they fall into a different group when they report back to the WSUS server by editing the "client-side targeting" entry and giving it a name like "Non-Domain" or something like that.

           

          Like I said, this is just my 2 cents - take any or all of it as you like.

           

          --KMSigma

            • Re: Re: Best Practices for "Catching up a Windows Environment" for patches.
              sinjen

              KMSigma,

               

              Thanks for taking the time to give me your thoughts. I do realize I gave very little information about the environment but wasn't exactly sure what would be relevant. My question regarding GPO's was too vague I see now. What I meant to ask is, do most of you have WSUS settings that allow the client wsus agent to schedule and install updates vs. setting that schedule up in SPM. My thoughts were ,for servers, that I'd use my WSUS GPO to push wsus settings to do a download and notify, then use SPM to schedule everything. This method seems like it would be much more flexible? I could do the same with multiple GPO's I suppose, but I don't see the point with the scheduling capabilities of SPM. Workstations on the other hand, I think I will just let fly (after a small pilot group of course).

               

              The issue of superseded updates is still very perplexing. After running WSUS reports yesterday and today, I see several updates that are "needed". After a couple hours running them all down, many of them appear to roll up to a Superseding update that has been PULLED by Microsoft. (KB2949927). I suppose if I had already applied the superseding updates prior to 2949927 coming out, I may never have known this had happened. But because we were behind, the older updates weren't approved and are showing as needed. They still haven't been approved because I've only been releasing updates that either have no superseded updates or are the latest version of a superseded update (just as you described). This is but one example that I've come across that's driving me nuts. Ignoring the network and disk affects, would there be any reason I shouldn't just go ahead and approve all "Needed/Failed" updates to get everything as up to date as possible?

               

              Your comments specifically related to Exchange make a lot of sense and is definitely the way we've decided to go.

               

              I also apparently don't understand the concept of the Credential Ring. We have a couple types of authentication here (like most environments I'm sure). We have a domain and most servers and workstations are member machines. We also have stand alone machines not part of our domain, therefore local credentials have to be used. Up to now, I have not been able to configure SPM to be able to authenticate to these machines using their local admin account. Note that when necessary, I have created a new local account and added it to the local admin group on the machine. So I do have a single local account I can use across the board for non domain machines. I just haven't been able to get it to work. Any pointers on use of the Credential Ring would be great. Perhaps I'm misusing it, I don't know.

               

              One final thought. Has anyone come up with a workaround to the age old issue of update prerequisites? Currently, If I have a new machine (that has a wsus gpo applied that downloads and schedules installation of updates at 1am), it can take several iterations (usually 3 to 4 days) for all patches to automatically apply. If I want all patches to apply during our scheduled maintenance window, I have to log on to the server and manually run the update checks and installations (until there are no more updates to apply). This is important because we only have small windows for reboots to occur. So, I'd love for all updates to be applied and all reboots happen on the same night automatically. Is there some technique, trick, workaround, creative scheduling in SPM that can help with this situation?

               

              Thanks again for your time.

               

              -Bo

                • Re: Re: Re: Best Practices for "Catching up a Windows Environment" for patches.
                  KMSigma

                  Regarding scheduling of updates - that's entirely up to you.  There was an awesome post (https://thwack.solarwinds.com/docs/DOC-175651) just made by travis.fenton.41 which calls out how to handle these schedulings if you want to utilize Patch Manager.  I know that you saw it!

                   

                  Regarding your KB2949927 difficulty, it looks like this falls into the "This #*$&@ patch broke my computer" file.

                   

                  Microsoft's latest on that KB is this: http://support.microsoft.com/kb/2949927

                  The security update that is described in the security advisory was removed from the Download Center because of an issue with the update. Microsoft is researching this problem and will post more information in this article when the information becomes available.

                   

                  In the most technical sense, this update has been expired by Microsoft.  The PowerShell script that I called out above, which runs after each synchronization, declined this update for my environments specifically because Microsoft "expired" it.

                   

                  Regarding Credential Rings: you can use a local admin account for your non-domain devices - you just need to define it differently.  I've edited the "default" credential ring and added a ".\Administrator" account to it.  Then I assigned that particular account as a Local Admin (Second Option).  This means that whenever Patch Manager tries to contact something, it will start with the Local Admin account.  If that fails, it will try the next level (or the <Default> if you have that defined).  Technically, this means that your on-domain devices will also be attempted first with the ".\Administrator" account as well.  That's ok in most environments because either the Admin is disabled or it's been renamed.

                   

                  Specifically on your final thought, the proper answer is to keep running the updates until there are no more.  I'm sure that in a perfect world there would be a time and place where you could install patch "B" along with security roll-up "A" for a specific product, but the detection algorithm that works within the Windows Update Agent isn't flexible enough to handle this at present.  The best way to avoid it is to have the systems regularly run the updates.

                   

                  I hope that I haven't muddied the waters for you any more than is strictly necessary and I hope that it helps you out.  If you want to speak more on this, feel free to contact me via direct message and maybe we can look at some stuff together.

                    • Re: Best Practices for "Catching up a Windows Environment" for patches.
                      sinjen

                      Yeah, I guess the problem I'm really having is that we are behind in patches so in the scenario of KB2949927, the superseded patches were not currently installed. So when MS pulled 2949927, it did not re-elevate all the patches that it superseded. So if you are only releasing based on the supersedence column and only releasing patches that have not superseded patches or the latest versions of superseded patches, none of the "needed" superseded patches roll out. UGH. So for now, I'm meticulously going through and approving any patch that's "needed" even if it has been superseded. Thanks for your advice. I'm sure once I get to the actual automation piece of this (automating the install of patches on a schedule), I'll have more questions.

                       

                      -Bo