26 Replies Latest reply on Jul 14, 2014 7:14 AM by tcbene

    Do You Trust Vendor Patches?

    Tom Hollingsworth

      Microsoft had a bit of an issue with the latest round of patches last week.  Nothing major blew up other than an Outlook folder pane view, but the implications were legion.  Do you actually test these patches before you deploy them?

       

      Microsoft isn't the only culprit, though they are the most visible.  Application vendors, operating system vendors, and even game companies have been bitten in the past about rushing incomplete patches out the door and causing more trouble than they tried to fix.  That's one of the reasons why many companies have started using the model of one patch day per month.

       

      Patch management software is critical in this regard.  Robust patch management allows you to designate a testing environment to ensure that things won't crash when you apply the latest security fix.  It's important to provide an additional level of confidence because your environment could be considerably different than the reference design.

       

      Vendors can't test everything.  There is a limitless combination of programs and file versions that exist on any number of systems in a typical enterprise.  All a vendor can do is ensure the patch won't destroy the base program while fixing the problem.  It should be up to the admins and technicians inside the customer environment to test the patch with the mix of software and hardware that they have installed.

       

      Good patch managers will give you every opportunity to use good practices to keep your systems from crashing when you deploy a new set of patches.  Using those features is up to you.

       

      Have you had a patch go wrong?  Did you test it first or did you rely on the vendor's assurance it was okay to install?  If so, are you going to test those patches in the future?

        • Re: Do You Trust Vendor Patches?
          derekschauland

          I have trouble on both sides of this issue.  Being a single person IT shop makes testing virtually impossible (since I do rather like to leave work) but blindly trusting vendor patches doesn't seem right either.  With the imaging tools and other things available today, I can rebuild a computer fairly fast (minus the Windows Updates) which takes less time than adequately testing every patch released.  Sort of a double edged sword I think

          • Re: Do You Trust Vendor Patches?
            wbrown

            Trust but verify.

            I trust vendor patches but if I see issues occur after installation I have no problem removing those patches.

            I used to not even question vendor patches but NT4.0 service packs cured me of that.

            • Re: Do You Trust Vendor Patches?
              cahunt

              NEVER! As Life's experiences have taught me to Trust No One! ... except maybe those guys on thesolarwinds-admin & those Funky Cats at Loop1 of course.

              Testing is key, though I know we can't all have fancy Test environments to stage the next patch, but it sure would be nice. I do agree with the Idea of regular backups, and an image capture before you do a major patch/upgrade just like derekschauland mentioned.

              But if this is not all available, wait at least 3 days and watch the vendor's forum's closely for any other user issues; Google for known bugs with the release... then again if it is a security feature you are getting with the update, make sure you have all the critical services in APM and the appropriate alerts setup if they stop or hang.  Tell your counterpart about it, and go fishing.

              • Re: Do You Trust Vendor Patches?
                rharland2012

                We have a small test group that we use to vet our WSUS stuff with good results.

                Strangely enough, the only real clunker we've had was a purpose-built app from one of our contract devs....it had a bad version of a system dll it replaced and bluescreened every computer it was installed on. While it was very stressful at times, the eureka-ness of the moment when we isolated root cause was a good reminder of why good troubleshooting is so very satisfying.

                • Re: Do You Trust Vendor Patches?
                  syldra

                  If they finally hire someone to help me here, then I'll set up a test environment because we've had problems in the past. But as long as I'm solo, it's trust for me here. As derekschauland said, I can rebuild a computer faster than I can test vendor patches. Of course, only my time is taken into account. If a patch goes wrong, and say 3-4 people are affected, the time wasted by all adds up to more than the time I would have wasted testing. It's a trade-off. More of mine or more of theirs.

                   

                  Right now, it's better more of theirs.

                  • Re: Do You Trust Vendor Patches?
                    broeben

                    I work in a small shop.  What we do is used phased patching.   So Week 1, we would roll out the patches to non-production low priority systems first.   Verify functionality.   If no issues, we would continue we weeks 2-3.  Higher priority servers would be scheduled for last.

                    • Re: Do You Trust Vendor Patches?
                      Aaron Denning

                      doesnt matter if there homemade patches or vendor patches going to have issues with them at least if you use vendor specific you can go to them and they can help you find out the real issue.

                      • Re: Do You Trust Vendor Patches?
                        Kurt H

                        I do not think you ca fully rely on the vendor to say a patch is ok. Most of the time the patch does not fit your environment. I can not tell you how many times I have seen management say a patch has to be applied immediately without testing and it causes havouc with a local system. Every patch should be tested to some degree to be sure it is going to work in your environment being that the vendors do not have the same enviroment that you do. Always test everything before you apply it.

                        • Re: Do You Trust Vendor Patches?
                          byrona

                          Due to our limited patching resources (people), our testing method is basically rolling the patches out to our internal systems a week before rolling them out to our customer systems.

                           

                          Tom Hollingsworth wrote:


                          Vendors can't test everything.  There is a limitless combination of programs and file versions that exist on any number of systems in a typical enterprise.

                           

                          Speaking of this, in the last round of Microsoft Patches we found a patch that conflicted with some VPN software that required us to rebuild the entire network stack on all impacted Windows systems.

                          • Re: Do You Trust Vendor Patches?
                            gunner2510

                            I trust them, but verify that they will work withour special configurations.  I realize that we have many different systems that have unique builds and any patch from any company could break them at any time.  I try to have them built as a VM and then snapshot the system before patching if possible.  if they affect a database, I make sure the backup was completed prior to patching.

                            I trust the Microsoft patches more than most other third party app and i have not had that many Microsoft patches cause issues in our environment since we try to keep all configurations as generic as possible.  we have over 800 Microsoft servers and have not had one of them shut down by a Microsoft patch in the past 4 years.

                             

                            Key words, Trust, but verify!

                              • Re: Do You Trust Vendor Patches?
                                Aforsythe

                                It really depends on the vendor. We go through so many updates for so many applications. But we're low on resources, so our patching processes are slow 2-3 weeks out (We don't push patches) and luckily major issues are usually found by the time we implement here. If we did implement patch management software or an appliance, we would want to patch on our time frame, pick which servers/workstations get the patches and patch in phases.

                              • Re: Do You Trust Vendor Patches?
                                rig24

                                We have a full testing environment that allows us to test all patches before deploying. In fact, our CM policies dictate that any patch be tested and confirmed before putting it into production.  I would feel comfortable enough with Cisco and Juniper to deploy the updates without testing, but when it comes to OS patches, complications come up to often to just blindly put them out there.

                                • Re: Do You Trust Vendor Patches?
                                  Alen Geopfarth

                                  Trust is a really strong word. We have a small bank of test servers to apply patches to first. If nothing explodes, we roll to other servers with an eye to rolling back.

                                  • Re: Do You Trust Vendor Patches?
                                    aschmitt

                                    I don't until we run it in a test environment for a at least a week to see what effects it may have on the rest of our production environment.  Although you can only be so "careful" one way or another a user in any environment will find a way to break it!

                                    • Re: Do You Trust Vendor Patches?
                                      slackerman19

                                      Trust but verify. To be honest thank goodness I don't have to verify desktops (virtual or physical). Its our operating procedure basically 2 week to roast in dev, if nothing blows up then move to production.

                                      • Re: Do You Trust Vendor Patches?
                                        Jfrazier

                                        Patch management is a necessary evil these days. I've seen a single Mcafee update bring down an entire datacenter/company for several days.  I've seen breakfix patches for monitoring software break other aspects of the product or open up new issues. I've seen patches re-introduce a previous bug that had been fixed and patched.  I can't manage all the patching that goes on at my company, but for the ones that deal with my tools and systems I have a method that supports my madness so to speak.  For OS and system patches, we patch and reboot DEV servers first....and wait a week before doing PROD servers.  For product patches and updates we also follow the same philosophy to give us time to test and validate.  We can't do a full regression test, but we have a representative group monitors and alerts on our DEV server to allow us to validate that what is in PROD works with the new patch/update.

                                        • Re: Do You Trust Vendor Patches?
                                          VM Solutions

                                          For the most part, vendor patches have gotten much better in the past few years, however I agree it is the patch manager's responsibility to perform test on the patch prior to widespread application. And even with the most careful of testing, there is always a chance something will go wrong on a percentage of your user's computers due to the various differences of programs & applications installed. To help prevent this, I recommend not just a single layer of testing, but once deployment begins to have multiple layers (circles) with each one increasing the number of systems being patch. This way, you have a better chance of catching any one-off issues in a smaller group of systems and not your entire enterprise.

                                          • Re: Do You Trust Vendor Patches?
                                            sevier.toby

                                            Yes.  Microsoft's success rate with patches is 99.999%.  Since I don't run antivirus software, keeping Windows up to date part of patching vulnerabilities that viruses exploit.  I've been on a team that verifies Windows Updates and since then have never met any organization that does test them, either at all or properly.  It is always left to the ego of jerk IT admins to pick and choose which one's they will approve as if they are god.  You need to be liberal with approvals, identifying systems that are not to upgraded and define the manual updates (e.g. SharePoint, SQL, and Lync). 

                                             

                                            Get rid of WSUS and let em rip.  Stop preventing your network from being vulnerable and when an update does break something it's your fault or an outdated application.  Find the root cause and get at the vendor to update their stuff.  Microsoft invest billions into testing updates before they are released.  What do IT guys know that Microsoft doesn't already? 

                                              • Re: Do You Trust Vendor Patches?
                                                rharland2012

                                                Sometimes the business will *not* hear 'it's your fault and/or outdated application'. We do run WSUS but are on the liberal side with the desktop infrastructure. We also have a test group, which has paid enormous dividends over the past couple of years.

                                                If we 'jerk IT admins' ever DID think we were god, our business counterparts would soon relieve us of that delusion.

                                              • Re: Do You Trust Vendor Patches?
                                                esther

                                                I strong agree, these patches from Microsoft and others should properly tested before it is sent out.......

                                                • Re: Do You Trust Vendor Patches?
                                                  aaron.damyen

                                                  I have found that it isn't 'trust' in the software that we are testing for, but more so the potential for something to change.

                                                   

                                                  My users seem to have more problems when their cheese moves than if the servers go down.  Back when I was a Systems Administrator, stability was my main concern and patching was always a priority.  I just ran the patches, verified the program started and passed it to the app owner.  Now, as an App Owner (of Orion! I win.), I'm more concerned about differences in functionality and UI changes.  I could care less about any memory leaks or virus vector possibilities.  This means that the tests the Systems Administrators put the software through is drastically different than those of the App Owners.

                                                   

                                                  Initially, my list of tests were small.  Login, run a report, view some graphs.  But, for each bug that arose, a new test was created.  I now had to verify that previous bugs didn't happen again.  My list of tests grows almost daily.  Are the gauges in the correct order, is green still green, did Average Scheduled Job Delay increase more than 100%, does the group status rollup still evaluate properly, and so on.  The tests continue to get much more specific.

                                                   

                                                  Like slackerman19 and other said, Trust* (* but Verify).  And script, script, script.  Testing will never decrease, do what you can to never have to do the work again.  Users will always find something to complain about and may not be as tolerant to cosmetic changes as you are.  Test your functionality first.  If you find cosmetic changes, send out a notification.  Many will forgive you if you tell them first and especially if they didn't read the notice. 


                                                  And always read the darn patch notes!  Whether you're a system admin or an owner, it's you in the pot, not the vendor.

                                                  • Re: Do You Trust Vendor Patches?
                                                    weiss

                                                    Trust but Verifiy. I know it's not always possible, but i believe in testing and verifying all patches.

                                                    • Re: Do You Trust Vendor Patches?
                                                      1to1riskcontrol

                                                      Most of the time yes, I trust them. On critical production systems, I'm tentative about installing them. I like to test in a dev environment first. I've been burned a few times by IBM on AIX systems.

                                                       

                                                      Joe

                                                      Private Investigator in Oklahoma City

                                                      Trust, then verify

                                                      • Re: Do You Trust Vendor Patches?
                                                        sevier.toby

                                                        Yep.  The best part is when the break something.  This let's me do what I do best - identify root cause.  No one knows better what's good for the product than the vendor.  It's a cool day when I can ID a vendor patch related conflict and present it too support and get another KB Article written just for me, and everyone else.  People get it patched and stop running scared from your job. 

                                                        • Re: Do You Trust Vendor Patches?
                                                          tcbene

                                                          I have found rollup patches often miss something they claimed to have rolled up, so I will normally install the previous patches even if the vendor doesn't require it.  Usually they find the roll up problem after the rollup release requiring another patch in the future.