It's the big day. You've got your rollout plan in operation, tested your patches and sent out your communications, yes you’ve guessed it, it’s patch rollout time. Patch days seem to go one of two ways: they are either very quiet or get very busy. Walking into the office on a patch rollout day is always one where you dread to look to see if you have a voicemail waiting already as that's not usually a good sign. Managing the rollout and keeping track of how it’s going are very important and essential if you are going to spot any issues early. During a patch rollout you will want to have several checks during the day to make sure that everything is running ok.
When users have an issue with a patch their first port of call will be the helpdesk. Keeping in touch with your helpdesk during the day to see what the call volumes are like is always a good idea and helps to spot any issues early on. Typically calls will be where there is an issue with a particular patch or application.
If there are issues being reported then you will need the following information from your helpdesk:
- How many users are affected? (This is important because if you have 500 successfully patched machines and only 2 failures, it might not be patch related!)
- Is the issue occurring on any machines that don't have the patches applied? (Helps to identify if it’s actually a patch issue or system issue.)
- Steps for recreating the issue. (What was the user doing to get the error? The key is for you to be able to test it yourself on a patched and non-patched machine.)
- Has the issue occurred previously? (If it’s occurred before, how was it fixed? Again helps to prove if the patches are at fault or not.)
Based on this information you can then start to investigate further and decide whether a patch is actually at fault or if the issue is unrelated.
Hopefully though when you check with your helpdesk there won’t be many calls at all.
Many automated patch tools have reporting built in so that you can see the progress of your rollout and easily view how many machines are showing as successfully patched and how many have reported a failure. It's also important to check that your patch system is running ok. The last thing you want on a rollout day is for your patching system to fail so regular health checks are key. You may already have these automated but it never does any harm to manually check the system as well.
Most patch rollouts will not have any issues but there's bound to be the odd occasion when you may be asked to stop a patch rolling out or remove a patch. It’s important therefore that you:
- Have a process for stopping a patch rollout. This may involve changing the status of a specific patch or re-creating the rollout without the affected patch.
- Have a process for removing a specific patch. Most patch management tools have an option to allow you to remove a previously installed patch. If yours doesn't then a script to run the removal command line of the patch would work or you could remotely connect and remove it.
- Ensure that you have good evidence that a patch is causing the issue. There are occasions when people will panic and want a patch stopped before you’ve even looked at it. Taking the time to go through the helpdesk checks above will ensure that you have tracked the issue to the patch rather than it being an unrelated issue.
As I said in my first blog, Patch Planning: Steps for an Effective Patch Management Process, patching is never straight forward but hopefully you'll find that you have more successful patch rollouts than you do issues.