30 Replies Latest reply on Oct 3, 2013 2:51 PM by scott.williams

    Patch Management? I don't care for it but YOU should!

    Carlo Costanzo

      Patch Management!  How exciting?  Eagerly waiting up till the wee hours of the night on Patch Tuesdays, waiting for Microsoft to release the latest round of security patches and application fixes.  I’m sure it’s like Christmas Eve EVERY SINGLE MONTH for Systems Administrators!

       

       

       

      As a consultant though, who coincidentally doesn’t really care much for the holidays either (all the hustle and bustle of people shopping and deadlines for purchasing things – Bah, Humbug!), Patch Tuesdays don’t even raise an eyebrow for me.  You see, I’m a project based consultant.  I normally enter environments with a specific purpose and specific deliverable.  Set up a solution, configure it, test it, document it, train the staff on it’s operations and then move on to the next project.  When I implement a system, it is normally completely patched up with the latest build numbers, versions and security fixes. Honestly, patch management is not even on my radar (or in my scope).  Sure, I know it will need to be done eventually, but it most likely will not be done by me…  Yeah, maybe I’m a rotten consultant but I think I am much more like a typical consultant on a typical project at a typical client in a typical environment.

       

       

       

      I have seen shrinking budgets that have pushed patch management to the bottom of most client’s priority lists.  I think there is a misconception that you can just run Windows Update on your machines and keep them up to date.  Of course if you manage 2 machines, go for it!  But as you scale up, you DO need a patch management solution to keep your systems up to date and secure.  As a consultant implementing solutions for clients, there is a reason that I am using the latest releases with up to date hotfixes and security patches.  It helps ensure that the solution will not only be its most reliable and stable but also that software vendors will be able to support the solutions efficiently.

       

       

       

      So I’m curious, when guys like me walk out of the building, are you (the client) putting in patch management solutions, clicking Windows Update every so often or just moving onto the next project (like me)?

       

       

      *Reply to this post to earn 50 points and 1 entry to win an iPod Nano

        • Re: Patch Management? I don't care for it but YOU should!
          bsciencefiction.tv

          We have a cutom built SCCM solution.  Machines not up to a cetain level are kicked off the network until they are updated.

           

          Funny story though.  We divested one of our holdings with approximately 150 PC's and about 15 Servers.  We had a patch management solution in place and we pick and choose our patches.  The company who bought this holding was a let Windows Update do that.

           

          After we removed our proprietary software and other security, they turned on Windows update...on all 150 machines and 15 servers at the same time.  With in 20 minutes they had crashed the network and clogged the Wan.  It was like someone trying to drink water from a fire hydrant through a straw.

           


            • Re: Patch Management? I don't care for it but YOU should!
              Carlo Costanzo

              That's funny about the mass downloading of patches.  That's definitely another benefit of centralized patching.  So on your home grown app, you would kick servers off the network as well?  It poses a chicken and egg deal doesn't it though?  If I'm OFF THE NETWORK, how can I get my patches?  or did you quarantine them to a 'dirty Network' until patched that had internet access?

               

              Dirty nets always intrigued me.  They remind me of real life Prisons.  You put a petty thief in there (missing a couple of patches) and before you know it, they are hardened criminals (completely compromised due to the honey pot nature of a dirty net).

            • Re: Patch Management? I don't care for it but YOU should!
              superfly99

              I don't care much for it either

               

              Another section of the IT department looks after patch management. They use SCCM to send out patches. They decide if and when patches are released to all machines.

              • Re: Patch Management? I don't care for it but YOU should!
                byrona

                We just recently implemented a patch management solution to replace WSUS.  We are a Cloud Solution Provider and WSUS didn't scale well or provide the flexibility we needed, especially in a non-domain environment.  I find it funny that just about every solution that Microsoft imagines assumes you will be running all of your systems in a domain, in the managed services world we have found that this often is not the case, at least not for smaller environments.

                 

                We ultimately settled on an agent based patch management solution that provides all of the flexibility and diversity that we required.  What this has given us is the ability to patch systems.. theoretically at least.  Now we have all of the issues associated with patching just on a much lager scale as patch management systems still don't fix the issues commonly associated with patching, at least not with Microsoft patching.  You still end up with some patches that just don't work, patches that conflict with customer software, SQL patches that require manual installation, etc, etc, etc.

                 

                My ultimate conclusion is that patch management systems are great at letting you manage your patching on a much larger scale with a much higher level of flexibility; however, they don't solve the problems typically associated with patching.  Patching remains a huge headache for us system administrators, especially in an extremely diverse environment.

                 

                If somebody out there has found a system that is contrary to my experiences I would love to hear about it! 

                  • Re: Patch Management? I don't care for it but YOU should!
                    mdriskell

                    I used Kaseya for patch management in a former life and as Byrona said it was a nightmare with diverse systems.  I am no longer responsible for that arena and I'm very happy for that fact.

                    • Re: Patch Management? I don't care for it but YOU should!
                      Carlo Costanzo

                        You still end up with some patches that just don't work, patches that conflict with customer software, SQL patches that require manual installation, etc, etc, etc.

                       

                       

                      Manual patches or hotfixes are a real pet peeve of mine.  I work a lot with Citrix Provisioning Server and many of their hotfixes are just text files with a bunch of instructions to copy out dlls and things.  It drives me MAD that they don't package it up into an EXE and force me to script it out...

                       

                      AND I COMPLETELY get the hypocrisy in my being upset over the programmer's laziness.

                      • Re: Patch Management? I don't care for it but YOU should!
                        Lawrence Garvin

                        > We are a Cloud Solution Provider and WSUS didn't scale well or provide the flexibility we needed, especially in a non-domain environment.  I find it funny that just about every solution that Microsoft

                        > imagines assumes you will be running all of your systems in a domain, in the managed services world we have found that this often is not the case, at least not for smaller environments.


                        Agree, WSUS is not well suited for implementation in a managed services environment, when the WSUS server is hosted by the MSP. Possibly why it's not licensed for use in that realm. :-)


                        The best way to approach WSUS in a managed services environment is to deploy independent WSUS servers at each customer's site (thus avoiding the need for a Microsoft SPLA). Of course, this is where you encounter the domain membership question for the remote console connection, as otherwise WSUS is a domain agnostic environment. The authentication of the remote console, however, is a security requirement -- and yes, for about the past ten years, Active Directory is the authentication mechanism for a Windows network.


                        A solution for the remote console challenge in this scenario, is SolarWinds Patch Manager -- which is exceptionally adept at allowing the administration of multiple WSUS servers in a single console, without the requirement of common domain membership (or domain trusts) between the console system and the WSUS Server.


                        > Patching remains a huge headache for us system administrators, especially in an extremely diverse environment.


                        Absolutely. I think much of this is because business management, and sometimes even IT Management, are critically unaware of the implications of patch management, the efforts necessary to properly implement a functional patch management solution, not to mention the risks of failing to properly implement a functional patch management solution. It's not easy. Assigning these tasks to junior sysadmins, which is quite often the approach of many organizations, just complicates matters even more, because, more often than not, those junior sysadmins do not have the cross-discipline skillsets necessary to fully appreciate the patch management process, much less properly implement a patch management product.

                          • Re: Patch Management? I don't care for it but YOU should!
                            byrona

                            LGarvin wrote:


                            Agree, WSUS is not well suited for implementation in a managed services environment, when the WSUS server is hosted by the MSP. Possibly why it's not licensed for use in that realm. :-)

                            So, it make sense that WSUS wouldn't be licensed for use in an MSP environment; however, that brings up another question...  If the SolarWinds Patch Manager runs on top of WSUS, wouldn't that mean that MSP's also can't use SolarWinds Patch Manager without violating the way WSUS is licensed to be used?

                              • Re: Patch Management? I don't care for it but YOU should!
                                Lawrence Garvin

                                A great question! and the devil is in the details of how WSUS is implemented, and the extra power that Patch Manager brings to a distributed WSUS implementation.

                                 

                                If the MSP hosts a WSUS server, and feeds other (customer's) WSUS servers or clients, this requires a Microsoft SPLA. It also has a whole bunch of security implications for "publishing a WSUS server to the Internet", but we can leave that for another thread.

                                 

                                If the customer hosts their own upstream WSUS server, and the MSP merely manages that server for them, then no SPLA is required, and the standard WSUS license covers this. The challenge, of course, as we've been discussing is establishing the WSUS console connection from a manager in the MSP office to the customer's WSUS server. It encounters the same "publish a WSUS server to the Internet" security complications, but also has to deal with the question of authenticating the console connection. Patch Manager eliminates the "publish the WSUS server" question, as well as the console authentication question.

                                 

                                What happens in this scenario is that a portion of the Patch Manager application infrastructure (the Automation Role) is installed on the customer's WSUS server (or any other system on the customer's site). This allows for a certificate-based encrypted and authenticated connection between the customer's WSUS server and the MSP's Patch Manager server, via the IANA-registered port 4092.

                          • Re: Patch Management? I don't care for it but YOU should!
                            RandyBrown

                            We use WSUS.  But, to be honest, we only patch our servers when there is a need.  Managing over 300 servers, each with very specific vendor approved patches/hotfixes/service packs along with very limited downtime windows does not really lend it itself consistent, regular updates.  That said, when we roll out a new server we do the same thing that you do ... get it as up-to-date as possible (drivers, patches, hotfixes, updates, bios, etc.).  Then, we too, move on to the next project and forget about updates until a need arises.

                            • Re: Patch Management? I don't care for it but YOU should!
                              mgarozzo

                              Like many others, we use WSUS.  In our case, we broke up servers into tiers based on importance within AD/WSUS which have different group policies applied.  Using a tiering method allows us to stagger updates throughout the month.  This works pretty well to give new updates a chance to run on non critcal qa/dev machines before being rolled out to the big boys.  It has worked pretty well thus far.

                              • Re: Patch Management? I don't care for it but YOU should!
                                mattoz

                                We use WSUS and patch monthly.  Occasionally we'll skip a month if there are no Critical patches, but there usually are.  We have proactive approach, so everything gets patched. We are looking to switch to SCCM 2012 later this year.  

                                • Re: Patch Management? I don't care for it but YOU should!
                                  chlsmith

                                  Try Shavlik, which was recently bought by VMWare and renamed to something else.   I used it for about 3 years in my last company and it worked flawlessly.  The best part was that the tasks occurred at the time they were scheduled reliably and it could patch a multitude of products beyond WSUS.

                                    • Re: Patch Management? I don't care for it but YOU should!
                                      Carlo Costanzo

                                      Yeah, they peeled it out of VMware Update Manager and rebranded it as VMware vCenter Protect Standard.

                                       

                                      VMware vCenter Protect Standard, Patch Management, Asset Inventory (Formerly Shavlik NetChk Protect)

                                       

                                      The coolest part of a Virtualization aware Patch Manager is the ability to patch both Offline and powered down virtual machines.

                                        • Re: Patch Management? I don't care for it but YOU should!
                                          byrona

                                          I really would love to have a patching solution that was VMWare integrated that was able to do pre-patching snapshots.  I know that these systems exist, unfortunately the one we have is not capable of this.

                                            • Re: Patch Management? I don't care for it but YOU should!
                                              Carlo Costanzo

                                              Yeah. that's the other neat feature of it.  Althought even more importantly, it REMOVES them!  Nothing worse than a 6 month old snapshot that know one has any idea what was for.

                                                • Re: Patch Management? I don't care for it but YOU should!
                                                  byrona

                                                  We actually evaluated that product before choosing the one that we have now.  I think the feature set of the Shavlik product was probably better in some ways; however, I am not sure I wanted to be even more "in bed" with VMWare than we already are.  That being said, the product we use does leverage Shavlik on the back-end.

                                                    • Re: Patch Management? I don't care for it but YOU should!
                                                      Carlo Costanzo

                                                      byrona wrote:

                                                       

                                                      That being said, the product we use does leverage Shavlik on the back-end.

                                                      Shavlik seems to be industry leader as a 3rd party patching source.  So many of the patch management companies seem to build the UI and then leverage Shavlik as the source.  Almost like a market feed of sorts.  I honestly can't even think of another 3rd party patch aggregator.

                                                        • Re: Patch Management? I don't care for it but YOU should!
                                                          Lawrence Garvin

                                                          Shavlik seems to be industry leader as a 3rd party patching source.  So many of the patch management companies seem to build the UI and then leverage Shavlik as the source.  Almost like a market feed of sorts.  I honestly can't even think of another 3rd party patch aggregator.


                                                          There are others, but yes, a  number of lesser products are built on top of the "Shavlik" engine, and continue to use content provided by the "Shavlik" catalog.


                                                          But the acquisition of Shavlik by VMWare two years ago, as well as the ownership of VMWare by EMC begs an ongoing question with regard to vetting the decision to buy a particular patch management solution based on "Shavlik" technologies: What is EMC/VMWare's commitment to continuing to license "Shavlik" technologies to those other vendors?

                                                  • Re: Patch Management? I don't care for it but YOU should!
                                                    Lawrence Garvin

                                                    The coolest part of a Virtualization aware Patch Manager is the ability to patch both Offline and powered down virtual machines.


                                                    Which actually brings up a question I've asked a couple of times before, and still warrants further discussion.


                                                    Which is better:

                                                    [a] blindly patching an offline VHD?, or

                                                    [b] powering on the VM and allowing software to apply the patches?


                                                    There's certainly convenience in the former, and powering on inactive VMs is not an ideal use of resources, but my concern with option [a] is that failures in the patch deployment process may not surface until the next time that virtual machine is powered on -- which may be many days/weeks later, and could even be after multiple patch cycles. Identifying the cause of a defective patch deployment on an active machine is hard enough; it seems pretty scary to me to try to imagine that on an inactive VM.


                                                    Having said that, it's worthy of note that SolarWinds Patch Manager does provide the capability for patching offline/powered-down VMs using method [b]. The Update Management Wizard in Patch Manager integrates Wake-On-Lan, update deployment, and System Shutdown activities in a sequenced and controlled fashion.

                                                      • Re: Patch Management? I don't care for it but YOU should!
                                                        Carlo Costanzo

                                                        LGarvin wrote:

                                                         

                                                        Which actually brings up a question I've asked a couple of times before, and still warrants further discussion.


                                                        Which is better:

                                                        [a] blindly patching an offline VHD?, or

                                                        [b] powering on the VM and allowing software to apply the patches?


                                                        There's certainly convenience in the former, and powering on inactive VMs is not an ideal use of resources, but my concern with option [a] is that failures in the patch deployment process may not surface until the next time that virtual machine is powered on -- which may be many days/weeks later, and could even be after multiple patch cycles. Identifying the cause of a defective patch deployment on an active machine is hard enough; it seems pretty scary to me to try to imagine that on an inactive VM.


                                                        Having said that, it's worthy of note that SolarWinds Patch Manager does provide the capability for patching offline/powered-down VMs using method [b]. The Update Management Wizard in Patch Manager integrates Wake-On-Lan, update deployment, and System Shutdown activities in a sequenced and controlled fashion.

                                                         

                                                        Very good points.  The Patch Management processes that I have used before that have the capability to patch offline machines ACTUALLY do it when the machine is online.  They typically put the machine into an isolated bubble, takes a snapshot, powers the machine on, pushes the patches to the server, applies them and then powers down and cleans up.  I've not run into any that surgically patch the VHDs directly.  That would be a pretty impressive (albeit scary) feat.

                                                  • Re: Patch Management? I don't care for it but YOU should!
                                                    xbod

                                                    We use a WSUS installation with a main server and then 2 servers at sites that have slower connections (those sites sync overnight from the main).  The main reason we have this in place is to limit internet usage.  Most of our computers get their updates from 1 of the WSUS servers depending on the site.  All other computers are done via Windows Update. 

                                                     

                                                    The policy is setup to control which server computers should use and to have the patches downloaded to the computers, but not installed.  We have techs that go out and process the updates.

                                                     

                                                    We have about 3 dozen servers that are done manually on a monthly basis.  

                                                     

                                                    The biggest problem is non-Microsoft patches.  JAVA, Flash, Adobe....  We've experimented with some patching solutions, but had little success so the techs do these manually.

                                                     

                                                    We have a policy that defines what our goals are as far as patching goes and these strategies meet those goals.  So for now, all is good

                                                      • Re: Patch Management? I don't care for it but YOU should!
                                                        Carlo Costanzo
                                                        The biggest problem is non-Microsoft patches.  JAVA, Flash, Adobe....  
                                                        We've experimented with some patching solutions, but had little success so the techs do these manually.

                                                         

                                                         

                                                        Those are the first ones I disable manually. Ha! Sometimes just removing the Pop up for the update is all you need to do to quiet a user.  It's such a psychological play to pop up an update alert to a user.  Once they see it, they 'begin' to have issues that needs to be fixed via updating.

                                                      • Re: Patch Management? I don't care for it but YOU should!
                                                        jsimon16

                                                        I used to use a tool called Patchlink and then the native vmware update manager until they took that away.  I agree the hardest challenges are non MS products.  I think MS is the best of the major companies to manage patches.  Every piece of softtware should have a solution like that.  To be honest desktops with a good GPO that keep autoupdating on are easy enough.  Servers are the only real challenge and again the MS servers like AD, Exch and SQl that are properly load balanced are not too bad.  It's the third party app servers that are not load balanced that need user downtime thats the real difficult part.  For a lot of those servers they may only get patched by us once a year.  But being 100% virtual like I am makes it safer in that I can always revert to a snap if it all goes wrong.

                                                         

                                                        Does anyone really see a performance boost from patches now a days?

                                                        • Re: Patch Management? I don't care for it but YOU should!
                                                          jeremymayfield

                                                          Of course you can use something like a Kace, LanDesk or other software, i stick to good old WSUS and have other application managed by IT for the users as we are a smaller shop, less than 100 PC's and can manage the other applications since there really are none outside of MS applications. 

                                                          • Re: Patch Management? I don't care for it but YOU should!
                                                            dave@entwistle.cc

                                                            We used to use WSUS and bad management of their clunky system has gotten us in trouble in the past.  We recently deployed SCCM and are looking for a major improvement.  Usually our issue is verifying that workstations and servers are relatively up to date on patches.  We usually find that the machines that are infected on the network are the most out of date patchwise. 

                                                            • Re: Patch Management? I don't care for it but YOU should!
                                                              storn

                                                              I believe most organizations today are running "something" for patch management"  We use SCCM to push OS updates only. We do not push App updates. OS updates are push regardless of time of day, so you get the friendly "reboot" message when your in the middle of that import project or config...

                                                              • Re: Patch Management? I don't care for it but YOU should!
                                                                scott.williams

                                                                We opt for fully patched BEFORE the consultant comes in.  That way we will know right up front how their solution handles it.  There is nothing worst than completing a project only to find that it breaks on the first update...  You'ld be surprised how many firms will jump on the "Blame MS" patch wagon...