Creating a standard patching process

The following points should be considered when creating a standard patching process.

Understanding the patch

First, know what the patch is for. Read the patch notes and details from the vendor and understand:

• what software or firmware is in scope
• what the dependencies are for deploying the patch
• what issues are resolved by the patch, and
• what the known issues are with deploying the patch. Any known issues with no workarounds could affect your business operations. That risk will need to be weighed up against the risk of not applying the patch.

Getting the patch files

When you get the patch files, confirm that the files come from an appropriate source. Before downloading files from a vendor’s website, check the TLS/SSL certificate details and verify that the owner of the domain is the owner you expect it to be. Phishing attacks use spoofed websites or emails to convince users to download and run malicious files. Don’t download your patch files from sources that you don’t share an encrypted channel with - like a website served over HTTP or a file share over FTP.

Often patches are made available through mirrors, or servers that have copies of patch files that may be geographically closer to your server (which can help with download speeds). You should only use mirrors that have signed patch files, so that you know it has from the right and trusted source, or at a minimum a checksum hash.

After you download the file, run a checksum (like a hash check using a hash function from the SHA-2 family) over the file. A vendor should provide you with their hash. Compare the two to make sure you have the right file. If the checksums are not the same, there could be a problem with the integrity of the file. You should contact the vendor to check.

Patch window and timing

Finding the right time to apply updates is a balancing act. You need to find the right balance of ensuring minimal impact to service (consider how you can patch without disrupting service at all), while also minimising disruption to your staff, and their personal time. Setting a patching window during a low traffic period will reduce the impact to end users. These windows should be long enough to allow the patching to finish. Consider the following things when setting the patching window:

• Allow time for a rollback in case the patching rollout fails later in the process.
• In case it is required, hold these windows at a time that will allow the component to be rebooted without massive disruption.
• If your company operates in multiple time zones and the low traffic periods are too short, you can consider multiple patching windows and deploying patches in batches.

Backup and rollback

Any time you are making changes to a system, it is important to have a rollback plan. This may involve taking backups before applying patches. Sometimes even after testing a patch successfully, there may be some bugs that are discovered later that disrupt your organisation. Having a backup to roll back to and having a clear way to make the call to roll back to a backup are important. The decision to roll back should consider:

• what the issue is
• if the vendor is aware of the issue
• when the vendor plans to have a fix, and
• if the impact is too high to the organisation to wait for that fix to be available.

Testing

First deploy a patch to a test environment or to a single instance before it is deployed to your entire environment. Test to make sure that the critical functions the software provides, still operates the same way it did before. This could be patching a gateway and testing to make sure that traffic still flows correctly, or patching a business application and testing to make sure the key features still work and produce the same outcomes. You can’t always test everything - it is usually a best effort attempt.

Notification to end users

Patching should be a regular process and most end users don’t need to know when it happens. To reduce notification fatigue, you should only push notifications to end users when emergency patches are applied outside of the regular patch window or when there is an action required on their part (ex: a system reboot). To encourage a culture of patching, it would be beneficial to have information and guidelines available to end users so they can learn more about how the process works, how the impact to end users has been considered (with areas like patch windows and test scope), and what vulnerabilities were recently patched.

Verification

After the patch has been deployed across the environment, it is important to verify that patching is complete. You could have received the notification that the patch file was pushed to a server, and the file was executed - however the server could have failed to reboot and the patch may not be in effect just yet. If the patch you deployed was widespread throughout the environment, you can perform this check by using active scanning tools to see if the vulnerabilities have been resolved or if the software has an updated version number.

Emergency

When a patch is released with no prior warning and outside of a normal vendor release cycle - that usually means it is critical. You should be prepared to review these type of emergency patches and determine if it is applicable to your environment. If the answer is yes, you should take the patching process that you have now and make changes to it that cuts down the time to deploy and adds to end user notification. Depending on the severity, you may decide to limited testing and instead notify end users of the severity of the issue and thank them for understanding that their use of the software may be interrupted while patches are rolled out.

Other patching content

Implementation advice for patching