4 Replies Latest reply on Jun 7, 2012 12:20 PM by antwesor

    I love the sound of breaking update windows in the morning

    Nonapeptide

      I fear patching. It is one of my biggest professional terrors, second only to managing backup and restore operations. Interestingly, patching is often intimately related to your backup and restore preparedness because patches often totally wreck a production system and require a hasty rollback or restore.

      I don't care what it is I'm patching, I hate it. Updating router firmware? I hate it. Flashing server BIOSs? Hate it. Deploying the once-a-day Adobe Reader and Java updates? Hate it with berzerker rage. SharePoint service pack? I think that falls into the category of "I'll change careers once I regain consciousness."

      Just last week, I had a network device that needed its firmware updated. The update conveniently reset the device back to factory settings! Fortunately, I meticulously document every choice in my appliances and other devices so bringing it back to normal was a fairly quick procedure. (I know what you're thinking: "Why didn't you take a backup of the firmware before updating it?" to which I answer: "DERP.") However, I was also blessed that the network device wasn't mission critical. Had it been a core component, things would have been very different.

      Right now I do things pretty vanilla. My Microsoft servers are patched with WSUS and my Linux servers get patched from the distribution's official repositories. Network appliances get patched when important bulletins are noticed, but there's no set schedule and if it's not a security patch, then firmware updates are likely to be ignored for a long time. Users' application patches are a nightmware since I currently deal with small offices. I've been wondering how I would handle larger fleets of servers and appliances (and desktop PCs as well) that required more carefully managed patching and updates.

      What do you use for patch management? How many different patching systems do you have? Separate ones for user applications versus desktop OS patches versus server OSs versus firmware patching? Or do you have an all-in-one solution? Do you have a set schedule for certain patches? Do you have well laid out rollback plans?

      I know I'm posting this in SolarWind's patch management section, but don't be shy about speaking of another product. If you use Lumension or Secunia or a Dell KACE appliance, share it all. Personally, I've been pulled in the direction of BigFix for OSs (but not so much after IBM inhaled it) as well as, yes, SolarWinds Orion for my network devices. I'd love to hear your opinions!

        • Re: I love the sound of breaking update windows in the morning
          KMSigma

          Our company uses Altiris for Desktop & Server Patch management.  However, there are several systems on which we do not do a monthly patching process; those being Exchange (regardless of version), SQL Server, and Domain Controllers.  We have a separate patch management process (but the same software) to handle the Domain Controllers.  Exchange is handled exclusively by the Exchange Team (of which I am a member) and SQL by the SQL Services team.  These include Windows updates.

          You are totally right regarding the nightmare that is software/firmware updates on network equipment.  We run dozens and dozens of network routers, switches, WAN Accelerators, Wireless LAN Controllers, and a plethora of other devices.  I've come up with a unique way to handle this that it might be worthwhile sharing.

          First and foremost, buy and use Solarwinds Orion Network Configuration Manager.  This single piece of software has saved our bacon more times than I'd actually like to admit.  It has come a long, long way in the past three years (since we picked up the product to supplement our other Orion products.

          Secondly, create yourself a "software deployment point" on a couple servers around your network.  We use a DNS Global Site Selector, so we can point the same URL to different servers depending on "source" of the request.  On our servers, we have HTTP/FTP/TFTP setup on one IP for writing and FTP/TFTP setup on a second IP for uploading.  I put a really simple web page in front of everything so that you could "browse" and then "copy" the links to paste in a command line window for an upgrade.

          Backing this set of servers (one in each data center), I have Windows Distributed File system.  This way, I can put a file on one server (like a new approved router image) and within a few minutes, it's available on the other server.  This means that if I try to download "ftp://deployment/software/3800/image-file-name.bin" from any device on the network, it finds the closest server one and pulls the "image-file-name.bin" file down.  This is especially useful when you deal with slow link speeds to satellite offices.  If Data Center "B" has faster connectivity to Branch "D", then pulling the image file from there would be preferable.

          I'm now in the process of working on standardizing our images for devices based on the role they play on our network (Core Switches, Edge Switches, WAN Routers, Voice Routers, WAN Accelerators, etc.), and I think that I'm going to use a Microsoft SharePoint 2010 workflow to handle that.  That's still in the air, but it's probably going to be a while until that particular thing sees the light of day.  For now, we'll just go through and do them in a phased way.

          I also have a custom Orion Report which pulls information directly from the Solarwinds Orion Database which pulls what I consider to be very important information.  You can also run it from SQL Management Studio.  This provides just about everything that I need to keep up to date on what software is running where.  There is also a similar standard report in Orion, but I like the simplicity of the SQL Query.

           

          [-------------------------------SQL Script-------------------------------]

          USE SolarWindsOrion

          SELECT  [Caption] ,

                  [IP_Address] ,

                  [DNS] ,

                  [SysName] ,

                  [Vendor] ,

                  [Description] ,

                  [Location] ,

                  [Contact] ,

                  [IOSImage] ,

                  [IOSVersion] ,

                  [MachineType]

          FROM    Nodes

          WHERE Vendor = 'Cisco' AND IOSImage<>'' AND IOSVersion<>''

          ORDER BY IOSVersion, IOSImage, Caption

          [-------------------------------SQL Script-------------------------------]

            • Re: I love the sound of breaking update windows in the morning
              Nonapeptide
              Our company uses Altiris for Desktop & Server Patch management.  However, there are several systems on which we do not do a monthly patching process; those being Exchange (regardless of version), SQL Server, and Domain Controllers.  We have a separate patch management process (but the same software) to handle the Domain Controllers.  Exchange is handled exclusively by the Exchange Team (of which I am a member) and SQL by the SQL Services team.  These include Windows updates.


              THIS! It's nice to see that you treat critical systems with kid gloves. In my experience, any update that touches SharePoint and IIS needs to be scrutinized and your backups need to be checked. In fact, that reminds me... I'm holding off on SP3 for WSS3.0 because of some suspicions I have about it's reliability. Next week could be... fun

               

              Secondly, create yourself a "software deployment point" on a couple servers around your network.  We use a DNS Global Site Selector, so we can point the same URL to different servers depending on "source" of the request.  On our servers, we have HTTP/FTP/TFTP setup on one IP for writing and FTP/TFTP setup on a second IP for uploading.  I put a really simple web page in front of everything so that you could "browse" and then "copy" the links to paste in a command line window for an upgrade.

              Backing this set of servers (one in each data center), I have Windows Distributed File system.  This way, I can put a file on one server (like a new approved router image) and within a few minutes, it's available on the other server.  This means that if I try to download "ftp://deployment/software/3800/image-file-name.bin" from any device on the network, it finds the closest server one and pulls the "image-file-name.bin" file down.  This is especially useful when you deal with slow link speeds to satellite offices.  If Data Center "B" has faster connectivity to Branch "D", then pulling the image file from there would be preferable.

              .

              That is the single most legit means of storing and retrieving updates I've ever seen. slow clap

              And yes, is DFS not one of the coolest features that Windows Server can pull off?

              Sounds like you have your update management down really well. How much of your time is spent on patching? Sounds like it's almost a full time job. Did it take a lot of convincing of the management to allow you to spend so much time on this kind of infrastructure?

                • Re: I love the sound of breaking update windows in the morning
                  KMSigma

                  Sounds like you have your update management down really well. How much of your time is spent on patching? Sounds like it's almost a full time job. Did it take a lot of convincing of the management to allow you to spend so much time on this kind of infrastructure?

                   



                  We don't spend too much time setting up and patching the Network Equipment.  We review the versions and see what new ones Cisco has available about every two weeks.  If there isn't a "critical" update (or a version where we've seen a bug from a TAC Case) we generally patch once a quarter, with one medium sized office running as the pilot (so that it has the various hardware platforms).

                  Currently, I'm the only one handling it, so it's NOT a full time job (I have several other duties which take precedence).  Honestly, there wasn't too much time and/or management buy-in to get this working.  Like I stated earlier, we've had NCM for a number of years and it has saved our bacon.  After working with it in a single data center, I decided to sketch out how to set this up for a dual-data center configuration.  It was just a small logic jump over to that for us.  Tweaking out IIS so that I could run the HTTP, FTP, TFTP with directory browsing and not reveal certain files took longer, but then again, I've been playing with that technology for a while now as well.



              • Re: I love the sound of breaking update windows in the morning
                antwesor

                We have been using WSUS since the 1.0 days. I have upgraded the WSUS server several times. Rebuilding a new WSUS server seems like the best option when an update to WSUS comes along.

                We also use Patch Manager (Eminentware) for a very long time now. I think I have rebuilt at least 2 servers since we have used Patch Manager. Our client base is pretty small ~600 or so clients and servers. Patch Manager allows very granular patching and also creates very detailed reports if needed by management. Patch Manager also seems like a good tool to use for 3rd party products like Adobe Reader, Adobe Flash Player, Java, Quicktime, and other common applications that exist on workstations.

                 

                Well, that is my 2 cents.