cancel
Showing results for 
Search instead for 
Did you mean: 

Create a Patching Plan of Attack (PPoA)

Level 11

It goes without saying that patching and updating your systems is a necessity.  No one wants to deal with the aftermath of a security breach because you forgot to manually patch your servers over the weekend, or your SCCM/WSUS/YUM solution wasn't configured correctly.  So how do you craft a solid plan of attack for patching?  There are many different ways you can approach patching, in previous posts I talked about what you are patching, and how to patch Linux systems, but we need to discuss creating a strategic plan for ensuring patch and update management don't let you down.  What I've done is laid out a step by step process in which you will learn how to create a Patching Plan of Attack or PPoA (not really an acronym but looks like a real one).

Step 1: Do you even know what needs to be patched?

The first step in our PPoA would be to do an assessment or inventory to see what is out there in your environment that needs to be patched.  Servers, networking gear, firewalls, desktop systems, etc.  If you don't know what's out there in your environment then how can you be confident in creating a PPoA??  You can't!  For some this might be easy due to the smaller size of their environment, but for others who work in a large enterprise with 100s of devices it can get tricky.  Thankfully tools like SolarWinds LAN Surveyor and and SNMP v3 can help you map out your network and see what's out there.  Hopefully you are already doing regular datacenter health checks where you actually set your Cheetos and Mt. Dew aside, get our of your chair and walk to the actual datacenter (please clean the orange dust off your fingers first!).

Step 2:  Being like everyone else is sometimes easier!

How many flavors of Linux are in your environment?  How many different versions are you supporting?  Do you have Win7, XP and Win8 all in your environment?  It can get tricky if you have a bunch of different operating systems out there and even trickier if they are all at different service pack levels.  Keep everything the same, if everything is the same, then you'll have an easier time putting together your PPoA and streamlining the process of patching.  Patching is mind numbing and painful, you don't want to add complexity to patching if you can avoid it.

Step 3:  Beep, beep, beep.... Back it up!  Please!

Before you even think about applying any patches, your PPoA must include a process for backing up all of your systems prior to and after patching.  The last thing anyone wants to do is have a RGE on their hands!  We shouldn't even be talking about this, if you aren't backing up your systems, run and hide and don't tell anyone else (I'll keep your secret).  If you don't have the storage space to back up your systems, find it.  If you are already backing up your systems, good for you, here's a virtual pat on the back!

Step 4:  Assess, Mitigate, Allow

I'm sure I've got you all out there reading this super excited and jonesing to go out and patch away, calm down, I know it's exciting, but let me ask you a question first.  Do you need to apply every patch that comes out?  Are all of your systems "mission critical"?  Before applying patches and creating an elaborate PPoA, do a risk assessment to see if you really  need to patch everything that you have.  The overhead that comes with patching can sometimes get out of hand if you apply every patch available to every systems you have.  For some, i.e. federal, you have to apply them all, but for others it might not be so necessary.  Can you mitigate the risk before patching it?  Are there things you can do ahead of time to reduce the risk or exposure of a certain system or group of systems?  Finally what kind of risks are you going to allow in your environment?  These are all aspects of good risk management that you can apply to your planning.

Step 5:  Patch away!

Now you have your PPoA and you are ready to get patching, go for it.  If you have a good plan of attack and you feel confident that everything has been backed up and all risks have be assessed and mitigated, then have at it.  Occasionally you are going to run into a patch that your systems aren't going to like, and they will stop working.  Hopefully you've backed up your systems or better yet, you are working with VMs and you can revert back to an earlier snapshot.  Keep these 5 steps in mind when building out your PPoA so you can feel confident tackling probably the most annoying task in all of IT.

14 Comments
goodzhere
Level 14

Good read!  I don't many environments where you can keep everything the same.  At least not in environments of any size.  It sure would be nice to have nothing but Windows or Linux, but that usually isn't a real possibility. 

mcam
Level 14

VM snapshots are your friend for patching

network_defender
Level 14

Best to test on a few before full deployment.  I know this isn't always possible, but good practice if you can.

mharvey
Level 17

Very good information.  This will help us going forward when more light gets turned onto patching.

mchenry2677
Level 9

great info

jkump
Level 15

This was insightful information.  With almost every environment that I have worked in, the concept of standardization is present but the reality is usually far different.  Different *Nix hardware, software, O/S, and applications--patching of course can break any or all of those.  Then, the windows patching whether automated or manual.

I find that the biggest hurdle to gathering all the proper information about the actual patches.  Having a test/sandbox has been the most benefit in determing which and what patches are needed in a particular environment but they too take time to work through.

gregwstuart
Level 11

Couldn't have said it better my friend!

gregwstuart
Level 11

Agreed, testing is always good when you are dealing with patches.  You never know the effect a patch is going to have on your system. 

gregwstuart
Level 11

Thanks, I'm glad it helped. 

gregwstuart
Level 11

Chris,

I know what you mean, it is difficult to keep everything the same, but if it can be done, there's definitely value added.  Thanks for reading!

gregwstuart
Level 11

Thanks!

gregwstuart
Level 11

Yes, standardization is more often than not set aside.  It takes real organization to get it right, but it can be done.  Agreed on the sandbox for testing, it's almost necessary to have a test environment and with virtualization it has become very easy to set one up.  The important this is that your test/dev environment remain in sync with your production environment, you want to see the same effect and reaction in the test environment that you would see in production. 

cahunt
Level 17

Test, test, test; burn in a new image/update .. very awesome read.

clubjuggle
Level 13

Our Engineering group actually does a great job of this.