Patch Planning: Steps for an Effective Patch Management Process - Timescales, testing and communication
[Editor's Note: PatchZone welcomes Alan Wenlock. Alan is from the UK and works in the Security and AntiVirus field. He's been working with SMS and Configuration Manager for ten years, and is also a moderator at MyITForum.]
Patching is a topic that always sends people into a panic for one reason or another. Whether you’re a Service Manager, Technician or Helpdesk Operator the word Patch always brings uncertainty or nervousness. Within the Patching area it’s not always clear to people how critical the task is, unless you get hit by a known exploit, or how many issues you might encounter as a result of patching.
Once of the main questions when looking at patching should be "What is the impact if your system is taken down by a known vulnerability and offline for x hours".
It can be hard work to start implementing a patch process and there will always be opposition against it but once you have a process setup and running it’s amazing how slick you can get your patching process to be.
There are several steps to an Effective Patch Management Process, but this is not to say that these are the only steps, and a patch process from one organisation can be totally different from another organisation as it has to adapt and work around the systems you are supporting. If patching was easy then we'd all have the patches downloading and installing automatically and we know this is not the case so how do you plan your patching process
It's key to have at least 2 processes when you are looking at Patch Management planning:
a) The first process is going to be for a regular patch cycle, like for example Microsoft’s monthly patch cycle or Adobe's quarterly one. This process will have a set timetable of activities that you following each month/quarter. In my experience its always better to try and patch sooner rather than later so in this timeline we’d be typically looking at a week long process giving you time to complete your activities and have sufficient testing.
b) The second process is how you deal with out of sync patches that do not fall under your regular patch activity. What happens when, for example, Microsoft release a critical patch at the end of the month and the vulnerabilities are being actively exploited. You can't wait for your regular patch process to come around, although it's an option, so what happens in these cases. In the case of out of band updates you need to act quicker and I’d always try and get the patch tested and deployed within a 2-3 days to ensure that your endpoints are patched and not vulnerable to the exploit. In my experience it’s critical to protect as many of your endpoints as possible to start to reduce the risk to your client network(s).
This is one of those areas when you need to decide how much testing you do as part of your testing cycle. It's impossible to test all applications unless you are lucky enough to only have a small number to look after. With the two different timetables we mentioned above its clear that the out of sync timetable is going to have less testing time then the regular cycle, perhaps a day’s testing at best. It's important to test the patches on a set of test machines first to make sure there are no issues actually apply the updates or after the reboot. Once these basic tests are complete you need to involve your apps team. It may be that your required to test the apps rather than a specific team but it’s important to have some kind of document to show the tests that are going to be done. This then becomes your baseline for testing each month and helps to maintain a standard patch testing process.
How long you test the patches depends on how critical a system is but also you have to weigh up how critical the patch is. If the patch is released out of cycle then I’d usually consider only doing a day’s testing, whereas with a regular monthly patch you might allow up to a week to test. As I mentioned earlier every organisation is different but you will still need some guide to how long your testing phases will be.
With any process that involves more than one team you are going to need to ensure that communication is taking place and even if you are testing the patches yourself against your applications you still need to make sure people like your helpdesk operators and service managers are kept up to date on any patch rollouts and even your customers so they know when to expect a patch. This is a key area that doesn’t always get as much attention as it should and I’ve always found that this is an area that can help your patch process run more smoothly. Customers will always be the first to complain about patches but once you have a regular patch cycle and associated communications they will soon learn to expect it.
So we've covered the timetables, testing and the communication at a high level but we have really only touched on the initial few areas of the patch process so far and we still need to look at doing the actual patch rollout and then how to monitor the rollout, liaise with your helpdesk and deal with issues and the potential to have to back out a patch, but that's all for another blog or two.