38 Replies Latest reply on Jan 26, 2016 1:50 PM by joliphant

    Why do you even bother patching?

    Carlo Costanzo

      So there are a bunch of reasons people patch their systems.  In fact, there are probably more reasons people patch systems than why they wouldn't. (although there are some valid reasons NOT to patch things IMHO)


       

      So why do you patch your systems?   Seems like a pretty easy question except that the answers can be pretty varied and there is usually a bit of overlap.

       


      Security?  People patch all the time for security.  Unpatched systems are just WAITING to be infected or exploited. No?

       


      Application fixes?  Stuff gets borked. Vendors push out patches continuously to fix things that should have never made it out of beta testing.

       


      Support compliance? If you are having any issue and call support, after pressing 1 for English, you are almost immediately directed to update to the latest hotfixes and patches for the particular product.  'Licensing issues?  Patch and then we'll talk.'  It can be almost comical at times.

       


      Because? Some people just do it because they were told to.

       

       

      Personally, since I deal primarily with new systems as a consultant, I patch for Application Fixes and Support Compliance.  Keeping your systems secure usually falls under someone else (Namely the client).   I rely on Firewall, Security, even Network guys to keep the Internet baddies out of my projects.

       

       

      Oh and don't worry if you are silently thinking Because as your reason.  That accounts for about 99% of the time I click Windows Update on my personal laptop.

       

      Carlo

       

       

      *Reply to this post to earn 50 points and 1 entry to win an iPod Nano

       

      Message was edited by: Carlo Costanzo Don't want to leave a comment? Try the quick poll - http://thwack.solarwinds.com/polls/1081

        • Re: Why do you even bother patching?
          RandyBrown

          Application fixes are the main reasons that we patch our systems.  Security ranks as the #2 reason, although we have far fewer security concerns as we do concerns about fixing broken areas of the applications that we use.

            • Re: Why do you even bother patching?
              Carlo Costanzo

              Application fixes are the main reasons that we patch our systems.

              So with Application fixes, do you patch Proactively (Apply any patch that 'Fixes' something) or re-actively (Search out fixes of things users report broken)?

               

              I know plenty of clients that only do patching re-actively since the act of actually patching applications makes them uneasy (If it ain't broke, don't fix it kinda thing).

               

              It seems to be such a delicate balancing act between trying to do good and trying not to do harm.

                • Re: Why do you even bother patching?
                  mdriskell

                  Our systems are patched with security fixes only...this is mandated by our security team.  Even servers that sit isolated far from any real risk are patched and those that don't get flagged by our scans.

                    • Re: Why do you even bother patching?
                      mdriskell

                      Unless there is a known bug we need to fix I should say.

                      • Re: Why do you even bother patching?
                        Carlo Costanzo

                        Our systems are patched with security fixes only...this is mandated by our security team.  Even servers that sit isolated far from any real risk are patched and those that don't get flagged by our scans.

                        I think Security Patching is by far the easiest politically.    Since it is typically mandated, you really don't have to worry too much about regression or application testing.  Security tends to trump all in those situations.  From a policy standpoint, if there is a security vulnerability, you just patch it and deal with the consequences with very little exception to the rule.  In most all other situations, the patch would need to be evaluated and considered before implementation. 

                          • Re: Why do you even bother patching?
                            mdriskell

                            We have three zones of systems dev/test/prod so they are patched in that order...Test mirrors prod identically so any issues should arise there first.  This limits our risk.

                              • Re: Why do you even bother patching?
                                Carlo Costanzo

                                We have three zones of systems dev/test/prod so they are patched in that order...Test mirrors prod identically so any issues should arise there first.  This limits our risk.

                                You would hope so. But its great that you have a process in place though.  As a consultant, I have plenty of first hand experience of the horrors in getting users to test ANYTHING.  Dual systems and parallel work environments have as much appeal to them as a marble notebook and pencil sharpener.  I can NEVER get them to accurately test the new systems I try to implement.  I can ONLY IMAGINE the challenges around routine patching and regression testing.  I suspect that for most companies with dev/test/prod, you would really need a catastrophic error immediately after installation in order to catch something.

                                 

                                There's only so much time in the work day and those bits and bytes aren't slowing down any.

                      • Re: Why do you even bother patching?
                        Sohail Bhamani

                        So having seen many customer environments, I can say that the majority of larger companies tend to have some sort of patching mechanism in place.  It seems the most fluid portion of most environments are the user machines.  It seems the vast majority of folks should be patching user machines.

                         

                        As for servers, many companies who have machines down a few layers of firewalls tend to not patch critical servers as sometimes this could interfere with the softwares on those machines.  All of these customers I have seen so far do not provide internet access to these machines and have ample security configured around and thus feel they do not need to patch.

                         

                        Sohail Bhamani

                        Loop1 Systems

                        http://www.loop1systems.com

                          • Re: Why do you even bother patching?
                            Carlo Costanzo

                            As for servers, many companies who have machines down a few layers of firewalls tend to not patch critical servers as sometimes this could interfere with the software on those machines.  All of these customers I have seen so far do not provide internet access to these machines and have ample security configured around and thus feel they do not need to patch.

                            I hear a lot about cutting internet access to unpatched machines or in this case, restricting Internet access therefore reducing the need to patch as much.  Local threats shouldn't be overlooked though.  Machines on a network where SOMEONE has internet access are IMHO running the same risk if they are connected or not.  Most malicious programs that take advantage of security holes attack indiscriminately within one's network trying to overpower other machines in it's wake...

                          • Re: Why do you even bother patching?
                            xbod

                            The main reason for patching is for security and then bug fixes.  My biggest pet peeve is when a patch unexpectedly changes software behavior from the user perspective (unless that's necessary to correct a bug).  It can be painful trying to explain to a user why they have to do something differently because we patched the software. 

                            • Re: Why do you even bother patching?
                              bsciencefiction.tv

                              Security and .net are typically our biggest driving factors in patching.  We have a test/dev and Prod/DR environment.  test/dev is always done first to find an errors or breaks.

                              • Re: Why do you even bother patching?
                                superfly99

                                Like most, we patch for security reasons. These are done on a regular basis or instantly if the security flaw is a major one. Servers are also rebooted on a regular basis.

                                 

                                Applications do get patched for bug fixes but these are usually done by the application user unless of course the application needs to be patch to fix a security issue. Applications are patched for bug fixes as necessary. "if it ain't broke, don't fix it".

                                • Re: Why do you even bother patching?
                                  rulob

                                  Althou Im not in charge of the Patch department, they guy who does is the desk besides me. I asked him why he would patch anything, he said: "Well, basically security, the policies here are very strict when it come to IT Security" He did add that support compliance is also a strong reason.

                                   

                                  I like how each company is different from one another. And how we network managers sometimes have to be up to date regardin ALL operations regarding the IT department(as a whole)

                                   

                                  Regards

                                   

                                  --Raul

                                    • Re: Why do you even bother patching?
                                      Carlo Costanzo

                                      He did add that support compliance is also a strong reason.

                                       

                                      Yeah- Support compliance is such a funny one though.  It's such a stall tactic of support organizations to say 'Update and then we'll support you'.  You update, the problem still exists and then support does it's thing.

                                    • Re: Why do you even bother patching?
                                      byrona

                                      We pathch our systems primarily for security.  As a service provider we also patch our cusotmers systems as it's part of the management service that they pay us for.  Our general policy when it comes to updates (patching, software updates, firmware updates, etc) is to keep things current.  It also helps from a support compliance standpoint when we need to work with our vendors.

                                      • Re: Why do you even bother patching?
                                        th3cap3

                                        I have to say security is the main reason, I can't really on others to always keep the baddies out. As far as my personal PC's, I usually patch because I like to be on the latest drivers/updates and I hate that stupid nagging popup to update my system.

                                        • Re: Why do you even bother patching?
                                          ccie14430

                                          For me it depends on the platform. Typically upgrades of routers and switches occur mainly for bug fixes and sometimes for security reasons. I usually leave firewalls alone until something significantly changes with code versions (ex. Cisco ASA pre 8.3 and post 8.3). For WAN Optimization gear I like to upgrade more often because of the frequency of added functionality.

                                           

                                          Later,

                                          -chris

                                            • Re: Why do you even bother patching?
                                              Carlo Costanzo

                                              Typically upgrades of routers and switches occur mainly for bug fixes and sometimes for security reasons. I usually leave firewalls alone until something significantly changes with code versions (ex. Cisco ASA pre 8.3 and post 8.3). For WAN Optimization gear I like to upgrade more often because of the frequency of added functionality.

                                               

                                              Chris,

                                              For the network gear, ALL patching is manual right?  Are there patch management solutions for Hardware and networking gear?  Do you just scour the vendor pages for updates and tech bulletins?  The fragility and importance of networking tends to make me think that it is all a manual process.  Both the discovery and implementation of patches.  Hopefully I'm not showing my networking naivety.

                                            • Re: Why do you even bother patching?
                                              Richard Nicholson

                                              I see most people patching like many have mentioned above.  Usually Security reasons first and foremost and bugs after that.

                                               

                                              Being a Network Engineer by trade I love how Cisco support does this (which is true for most big vendors).  My favorite sequence of events when talking with TAC..

                                               

                                              Me:  Hi,  I have a issue with one of my switches.  It seems I can't get this Port Channel to correctly bind to the interfaces I tied it too, but the code I am using shows no issues with this feature on other equipment, and I have this configured on the same code version across 100 other switches... 

                                              Cisco:  Sorry to hear that you are having this issue.  I recommend you upgrade to the latest code re-vision, and this sending us a Diag file.

                                              Me: Really???  So because this doesn't work on only one device, and it's working on 100 other devices running the same code I have no choice but to upgrade before you will help me.

                                              Cisco:  Yes,  That version of code you are using is older and not the latest code, and we won't trouble shoot that version anymore.

                                               

                                              ARRGH!!  This is always so common with big vendors.  Upgrade your code and then we will talk.

                                               

                                              I know that was a bit off topic, but I thought it was funny how this is very common.

                                              • Re: Why do you even bother patching?
                                                jeremymayfield

                                                whether we like it or not, software will be updated.  Somewhere some day someone will need a new java or flash.  So its best to just accept what you can not change and find the best way to adapt the least amount of work and cost.  I think security is up there but more compatibility.   I update BIOS more than anything.  trying to stretch PC's longer and longer and windows keeps pushing drivers nd stuff that break my machines.   I find BIOS updates help the most.  So we have procedures now to ensure all BIOS gets updated when we put our hands on a PC to work on it or clean it.

                                                • Re: Why do you even bother patching?
                                                  Richard Nicholson

                                                  The biggest concern I see in patching is watching out for legacy applications that are problematic when a system is patched.  All to often a company is running a proprietary piece of software/code, or older version of a software platform that will easily be broken upon applying patches to their systems they run on.  This keeps them from wanting to patch because they don't want to expend the money/time to have the code/software updated, or fixed so the patch/security fix doesn't break the application, or the software company wants an insane amount of money to get back under maintenance and allow them to update.

                                                   

                                                  I have seen plenty of systems on an Internal network that haven't seen one patch, but they are some of the most locked down systems with IDS/IPS, Firewalls, Software Firewalls, and 2 factor auth sitting in front of them because of this.  I have also seen the opposite where the box is in a DMZ or Public facing and it's just begging to be cracked and zombie'd for use as a Spam server..

                                                  • Re: Why do you even bother patching?
                                                    shuth

                                                    For SolarWinds specifically, application/bug fixes are the primary reason however it is closely followed by additional features being added (and as you said, if you call support the first troubleshooting method is usually "update to the latest version"). The recent NPM 10.4 update had clients split 50/50 - some liked having custom properties managed via the GUI while others lamented the loss of the separate application on the server. The recent 10.4.1 update resolved a lot of the issues the latter clients had.

                                                     

                                                    In general though, servers typically get the security patches and my laptop follows Windows Updates suggestions.

                                                    • Re: Why do you even bother patching?
                                                      dave@entwistle.cc

                                                      My policy on patching usually is to wait until a patch either fixes a specific issue, allows a new feature I want to use, or is necessary for a compliance report I need to run.  I have found out the hard way that patching for patching's sake can usually cause issues because it will usually break any customization and sometimes finding the customization that was broken by the patch is difficult.

                                                      • Re: Why do you even bother patching?
                                                        storn

                                                        Because were told too

                                                        • Re: Why do you even bother patching?
                                                          Andrew M

                                                          We patch mainly for security, though at times I wonder if I'm just spinning my wheels. Lately it seems that by the time a security patch is released, the vulnerability has already been in the wild for weeks or even months. Testing is often minimal, due to time and resource constraints and I sometimes worry about applying a bad patch. Third party patches are the biggest time sink, due to less confidence in the publisher testing and our own package customization requirements. I am working to remove as many unnecessary third party apps as possible (i.e. Java), but I don't see any way out of patching so on it goes.

                                                          • Re: Why do you even bother patching?
                                                            fdporter54

                                                            PCI Compliance.  If your organization processes credit card transactions, or even just stores credit card data on your premises, your entire company must maintain compliance with the industry standards for "PCI Compliance".  Federal laws govern data security for credit card providers and processors - these requirements filter down to the companies that accept credit cards from their customers.  Policies, procedures and software designed to keep your systems secure and up to date can save your company from having to pay deposits with vendors like PayPal; failure to comply can increase your financial liability should your company's physical (ie, paper) or electronic storage systems get hacked.  A lawsuit for "failure to take appropriate security measures to protect customer credit card data" can damage your company's reputation and financial stability.  If you don't take it seriously, rest assured - some lawyer will (if you lose control of your customers' credit card data through neglect).

                                                            • Re: Why do you even bother patching?
                                                              bigk101

                                                              I consider the Patch process as mandatory ...Why ?... because if I didn't patch and update things , I can have absolutly no garrantee of security or sucess ! Not to mention the problems my infected or unpached or non updated machine could cause to other machines and or projects !

                                                              • Re: Why do you even bother patching?
                                                                richardteachout

                                                                In my opinion, Patching isn't an option, it's a requirement - although it's not often possible to be done without breaking applications.

                                                                That makes it a WIN/LOSE situation, doesn't it.

                                                                So, patching needs to be a balance between security teams, application teams, and executive teams - they all need to be on board, or it fails.

                                                                • Re: Why do you even bother patching?
                                                                  pyro13g

                                                                  Just some general stuff here.

                                                                   

                                                                  Patch out of need not for the sake of patching. 


                                                                  Depending on patch size and the number of items in the patch, you often trade one set of problems for another.

                                                                   

                                                                  There are some things we don't patch because we don't carry a maintenance contract on the gear.

                                                                   

                                                                  "Patch first, then call us back" is the last thing a company you pay for support should say.

                                                                  • Re: Why do you even bother patching?
                                                                    joliphant

                                                                    I run Windows Server 2012 on VMs and only patch for major security issues and application fixes, such as the last SMB issue that was bring down my home directory server.