Implementation advice for patching
Each environment is unique and can be complex. Here are some key considerations for patching in your organisation.
There are several things to think about when attempting to implement consistent patching across an organisation. The sections below point out the different steps you should be taking when it comes to performing patching.
1. Understand your environment
Understand what software and firmware you have in your environment, so that you know what needs to be patched. This allows you to know which patches are applicable to you and what the impact would be if a vulnerability is exploited. As a company grows in size, it gets harder to track what is actually within the environment.
For larger organisations, a configuration management database (CMDB) can be helpful and used to give you a granular view of the components in your environment - from the hardware and devices all the way down to the software versions they are running. CMDB are often populated by hand - which means they are often out of date, or simply have incorrect information. Having automated reporting to ensure that the CMDB is always up to date is very valuable. Having a central repository makes it easier for this data to feed into other processes, like patching or incident response.
Even without a CMDB, an organisation can still benefit from using tools that scan and collect data about the environment. Software discovery and inventory tools, such as software inventory in Windows System Centre Configuration Manager (SCCM), help you understand what software is running in your environment. Vulnerability scanning tools, such as Nessus or OpenVAS, help you identify which vulnerabilities your environment is currently exposed to. It also provides patch references for those that have been resolved by the vendor.
Scanning and discovery tools can only help when a device or hardware is connected to your network, and reachable by the scanner. The software status of segmented networks, or devices that are regularly taken off the network such as laptops and other mobile devices, can be hard to capture.
TIP: If staff are able to use their own personal devices in your environment (Bring Your Own Device / BYOD), you may be able to discover these devices but you will have little to no control over the software patches that are applied. Mobile device management (MDM) tools give you more control over these untrusted devices by enforcing security policies and patch levels. If you don’t use a MDM tool, have a clear agreement and policy with your staff around patching their devices. If staff don’t patch, be prepared to ring-fence these devices on a separate untrusted network until they are patched.
TIP: Hardware and devices may be in a position where they can’t be patched due to hardware limitations. For example, a Samsung Galaxy S III that can’t support the latest versions of the Android operating system, or a legacy application that needs to run on a Windows 2003 server. The legacy software on these devices and hardware won’t be able to be patched and pose a bigger risk to the organisation.
2. Know when updates are released
Vendors typically send notifications out to their mailing lists when patches are released. Some vendors have clear patch release days - like Microsoft’s Patch Tuesday. Once you have an idea of the software you are running in your environment, make sure you have a clear and central way to receive patch notifications.
Keep track of the vendors who have regular patch release dates and consider these release dates as part of your patching process. Vendors may release emergency patches outside the regular release date so it’s a good idea to sign up to their mailing lists and notifications.
TIP: Send these notifications to a central location to review and filter. This will prevent urgent or important notifications from being lost in the inbox.
TIP: For an additional step, you can subscribe to data feeds for vulnerabilities. NIST provide multiple data feeds, including an RSS feed. These provide updates on recently announced public vulnerabilities. The solution to a vulnerability may include an immediate release of a patch, so it can help to keep track of recent vulnerabilities to stay across emergency patches.
TIP: If possible, integrate this information with your CMDB, so you can automatically see which systems need to be updated.
3. Have a standard process for applying patches
You have to have a clear process for applying patches - sometimes patches will come with no warning and will need to be applied as soon as possible. Having a clear set of steps for each standard patch release, will make it easier when the emergency patches are released.
Some things you may put into the process are:
- understanding what the patch is for
- setting a time window to deploy the patch
- back-ups and when to do rollbacks
- testing the patch
- notifying end users
- verifying the patch is complete
- what to do when there’s an emergency patch.
See standard patching process for more detail on what to include at each step.
You can configure some software to automatically apply updates when patches are released. We recommend you apply automatic updates where possible. You need to balance the risk between applying a patch that may contain a bug against the risk of including those patches into your increasing patching cycle.
TIP: Automating the deployment and testing of patches can help you save time and resources - and also win you favours when it comes to discussing the risk of patching to the business. Consider using tools like WSUS or Ansible to automate downloading and applying patches, and running of scripts to test that patches were applied correctly.
TIP: Where possible, design and build your systems so that you can patch without outages. By ensuring that patching and rebooting one component will not cause a system outage, you enable your staff to apply updates during business hours, increasing the likelihood of patches being applied, and also ensuring that if something goes wrong, the right people are awake and on hand to resolve the issue.
If you don’t perform your patching process regularly and your software starts falling multiple cycles out of date, the risk becomes higher. For example, you may try to make your software current again by installing lots of updates. Patches aren’t just security fixes - they can also include changes to features and to the actual software experience itself. Because of this, updating several patches at once could have unintended impacts to end users or software performance. Applying more updates at the same time also increases the duration of the change window, and you may find that some updates require previous updates, and so you have to perform multiple rounds of updates to get completely up to date.
4. Highlight the risk of delaying patching
Despite patching being critical, it is often not done. There could be many reasons why - not enough time, not enough resources, too many other critical processes or issues being handled. Our recommendation is that all software should always be kept up to date, and organisations should only consider delaying patching in exceptional circumstances.
If your organisation is considering delaying patching a vulnerability, communicate with the business and risk owners so they understand:
- which system or components are directly or indirectly impacted - including the systems that the vulnerable system communicates with
- the risk the vulnerability carries - including what an attacker could achieve and what the likelihood of carrying out that attack is
- the impact to the organisation if the vulnerability is exploited - including downtime or loss of intellectual property or customer data
- the organisation’s ability to detect if the vulnerability has been exploited - or other controls that may prevent it from happening.
Based off this understanding, work with the risk owners to determine if the risk is worth delaying a patch. Make sure decision makers have all the information they need to make the best decision.
Keep track of all the patch exceptions the organisation makes. One decision may not have a high impact, however when looking at the aggregate of all the previous decisions, the risk or impact may be higher. Attackers are often able to exploit unpatched components and chain previous, older vulnerabilities to carry out an attack. It is important for the business to look at the whole instead of one patch on its own.
TIP: Explaining the risk can be difficult. Try explaining the different scenarios that could happen if a patch is not applied. If there are multiple patches missing, explain how unpatched vulnerabilities could be chained together and introduce higher risks. For example, a cross-site scripting (XSS) vulnerability on a webserver may not be considered high risk on its own. However this could be chained with other vulnerabilities, like open redirect vulnerabilities, to raise the risk higher.
When discussing the need for patching, sometimes people focus on preferring features to fixes. Try to explain that patches are necessary in order to get the ability to try new features. Security fixes can critically change the way software is used, and future features will be built on top of that new model. By falling back and not applying security fixes, this could prevent those new features from being used. It also prevents your IT team from focusing their time and resources on forward thinking tasks as they may have to spend more time monitoring the environment and monitoring those security holes.