Skip navigation

PatchZone

3 Posts authored by: awenlock

It's the big day.  You've got your rollout plan in operation, tested your patches and sent out your communications, yes you’ve guessed it, it’s patch rollout time. Patch days seem to go one of two ways: they are either very quiet or get very busy.  Walking into the office on a patch rollout day is always one where you dread to look to see if you have a voicemail waiting already as that's not usually a good sign. Managing the rollout and keeping track of how it’s going are very important and essential if you are going to spot any issues early.  During a patch rollout you will want to have several checks during the day to make sure that everything is running ok.

 

Helpdesk Monitoring:

When users have an issue with a patch their first port of call will be the helpdesk. Keeping in touch with your helpdesk during the day to see what the call volumes are like is always a good idea and helps to spot any issues early on.  Typically calls will be where there is an issue with a particular patch or application.

 

If there are issues being reported then you will need the following information from your helpdesk:

  • How many users are affected? (This is important because if you have 500 successfully patched machines and only 2 failures, it might not be patch related!)
  • Is the issue occurring on any machines that don't have the patches applied? (Helps to identify if it’s actually a patch issue or system issue.)
  • Steps for recreating the issue. (What was the user doing to get the error? The key is for you to be able to test it yourself on a patched and non-patched machine.)
  • Has the issue occurred previously? (If it’s occurred before, how was it fixed? Again helps to prove if the patches are at fault or not.)

 

Based on this information you can then start to investigate further and decide whether a patch is actually at fault or if the issue is unrelated.

 

Hopefully though when you check with your helpdesk there won’t be many calls at all.

 

System Monitoring:

Many automated patch tools have reporting built in so that you can see the progress of your rollout and easily view how many machines are showing as successfully patched and how many have reported a failure.  It's also important to check that your patch system is running ok.  The last thing you want on a rollout day is for your patching system to fail so regular health checks are key.  You may already have these automated but it never does any harm to manually check the system as well.

 

Backout Plans:

Most patch rollouts will not have any issues but there's bound to be the odd occasion when you may be asked to stop a patch rolling out or remove a patch.  It’s important therefore that you:

  1. Have a process for stopping a patch rollout.  This may involve changing the status of a specific patch or re-creating the rollout without the affected patch.
  2. Have a process for removing a specific patch.  Most patch management tools have an option to allow you to remove a previously installed patch.  If yours doesn't then a script to run the removal command line of the patch would work or you could remotely connect and remove it.
  3. Ensure that you have good evidence that a patch is causing the issue.  There are occasions when people will panic and want a patch stopped before you’ve even looked at it.  Taking the time to go through the helpdesk checks above will ensure that you have tracked the issue to the patch rather than it being an unrelated issue.

 

As I said in my first blog, Patch Planning: Steps for an Effective Patch Management Process, patching is never straight forward but hopefully you'll find that you have more successful patch rollouts than you do issues.

On paper patching can seem quite straight forward.  You have a system running that needs a patch so you just need to install the patch and job done.  Unfortunately in the Enterprise environment it’s never that simple and there are a lot of obstacles that will be put in your path.

 

As I mentioned in my previous blog there is never a "one solution fits all" for Enterprises and it’s important that you understand your own environment and tailor your patch management to meet your own specific setup.

 

In my experience there are a few key areas that I believe are important to any Enterprise patch process.

 

1) Having a complete Asset Inventory

Before you can start patching you really need to know your Enterprise setup and what systems you have.  The mix of Operating Systems you are covering and what applications you are responsible for keeping up to date is important.  You may have common applications, for example Adobe Flash Player, which you have installed on all your machines.  There are many patching and software delivery tools that will collect an inventory as well.  You can then use this inventory data to build groups of machines with an affected application or component so you can see how many affected machines you need to patch.

 

2) Ensure that your patch has been through Testing

Any patch that you look to roll out will need testing as the last thing you want is to create an issue when you roll out a patch.  Not testing patches can have a negative effect on your patch management process so it’s important that you have a good range of systems that you can test on, either in a lab or nominated users in your enterprise who are happy to test patches, and if something goes wrong are able to remove or re-install them.

 

3) Plan how the Deployment of the patch is going to be done

Once you are ready to deploy you need to decide how aggressive to be with the deployment.  If it’s a critical patch you may want to target all machines within a few days whereas a less critical patch may be split into batches and deployed over the period of a week. In my experience a pilot of between 200-500 machines before the main deployment always helps to build confidence with your Service Managers that a patch is deploying successfully.

 

4) Dealing with any Exceptions

There are always going to be one or more machines in your Enterprise that are either too critical to patch, have issues with a particular patch or who have a specific maintenance window.  In these cases you need to make sure that you have these exceptions not only documented but also reviewed on an on-going basis. It's quite easy in a large Enterprise to lose sight of your exceptions if they are not documented and these may end up posing the biggest risk to your enterprise if not reviewed on a regular basis.

 

5) Produce reports that show progress

Now that the patch has been deployed the process shouldn't stop at that.  Have you checked that the deployment is actually working as it should and that the patch you are deploying is being applied successfully?  Many Enterprises will ask for progress reports of any patch deployment so it’s important that you produce these to show not only the progress of the deployment but also how many machines have actually been patched.  This builds confidence that you have successfully patched the system and the risk reduced.

 

That's just a few areas that you need to consider when looking at best practices for your patch process and you will need to make tweaks and additions to meet your own specific setup within your enterprise, I never said it was easy.

Patch Planning: Steps for an Effective Patch Management Process - Timescales, testing and communication

 

[Editor's Note: PatchZone welcomes Alan Wenlock. Alan is from the UK and works in the Security and AntiVirus field. He's been working with SMS and Configuration Manager for ten years, and is also a moderator at MyITForum.]

 

Patching is a topic that always sends people into a panic for one reason or another. Whether you’re a Service Manager, Technician or Helpdesk Operator the word Patch always brings uncertainty or nervousness.  Within the Patching area it’s not always clear to people how critical the task is, unless you get hit by a known exploit, or how many issues you might encounter as a result of patching.

 

Once of the main questions when looking at patching should be "What is the impact if your system is taken down by a known vulnerability and offline for x hours". 

 

It can be hard work to start implementing a patch process and there will always be opposition against it but once you have a process setup and running it’s amazing how slick you can get your patching process to be.

 

There are several steps to an Effective Patch Management Process, but this is not to say that these are the only steps, and a patch process from one organisation can be totally different from another organisation as it has to adapt and work around the systems you are supporting.  If patching was easy then we'd all have the patches downloading and installing automatically and we know this is not the case so how do you plan your patching process

 

1) Timetable:

It's key to have at least 2 processes when you are looking at Patch Management planning: 

 

a) The first process is going to be for a regular patch cycle, like for example Microsoft’s monthly patch cycle or Adobe's quarterly one.  This process will have a set timetable of activities that you following each month/quarter.  In my experience its always better to try and patch sooner rather than later so in this timeline we’d be typically looking at a week long process giving you time to complete your activities and have sufficient testing. 

 

b) The second process is how you deal with out of sync patches that do not fall under your regular patch activity.  What happens when, for example, Microsoft release a critical patch at the end of the month and the vulnerabilities are being actively exploited.  You can't wait for your regular patch process to come around, although it's an option, so what happens in these cases.  In the case of out of band updates you need to act quicker and I’d always try and get the patch tested and deployed within a 2-3 days to ensure that your endpoints are patched and not vulnerable to the exploit.  In my experience it’s critical to protect as many of your endpoints as possible to start to reduce the risk to your client network(s).

 

2) Testing

This is one of those areas when you need to decide how much testing you do as part of your testing cycle.  It's impossible to test all applications unless you are lucky enough to only have a small number to look after.  With the two different timetables we mentioned above its clear that the out of sync timetable is going to have less testing time then the regular cycle, perhaps a day’s testing at best.  It's important to test the patches on a set of test machines first to make sure there are no issues actually apply the updates or after the reboot.  Once these basic tests are complete you need to involve your apps team.  It may be that your required to test the apps rather than a specific team but it’s important to have some kind of document to show the tests that are going to be done.  This then becomes your baseline for testing each month and helps to maintain a standard patch testing process. 

 

How long you test the patches depends on how critical a system is but also you have to weigh up how critical the patch is.  If the patch is released out of cycle then I’d usually consider only doing a day’s testing, whereas with a regular monthly patch you might allow up to a week to test. As I mentioned earlier every organisation is different but you will still need some guide to how long your testing phases will be.

 

3) Communication

With any process that involves more than one team you are going to need to ensure that communication is taking place and even if you are testing the patches yourself against your applications you still need to make sure people like your helpdesk operators and service managers are kept up to date on any patch rollouts and even your customers so they know when to expect a patch.  This is a key area that doesn’t always get as much attention as it should and I’ve always found that this is an area that can help your patch process run more smoothly.  Customers will always be the first to complain about patches but once you have a regular patch cycle and associated communications they will soon learn to expect it.

 

So we've covered the timetables, testing and the communication at a high level but we have really only touched on the initial few areas of the patch process so far and we still need to look at doing the actual patch rollout and then how to monitor the rollout, liaise with your helpdesk and deal with issues and the potential to have to back out a patch, but that's all for another blog or two.