A critical review of risk matrices used in water safety planning: improving risk matrix construction

Although some water utilities in Australia and elsewhere began implementing risk management and drinking water quality management systems in the mid-1990s, the Sydney water crisis of 1998 (Clancy 2000) provided the impetus for developing national guidance under the purview of the National Health and Medical Research Council (NHMRC) of Australia, launched in October 2000. This initiative was advanced by a joint meeting in Adelaide, Australia in May 2001 of the NHMRC working group (Rizak et al. 2003) with the World Health Organization (WHO), Water, Sanitation, and Health expert group on drinking water, led by Dr. Jamie Bartram. WHO had been working actively on similar concepts for drinking water since Havelaar (1994) had proposed adopting the risk management approaches adopted by the food industry. These initiatives resulted in the major restructuring of the Australian Drinking Water Guidelines in 2004 to adopt a holistic preventive risk management approach and similarly in the third edition (2004) of the WHO Guidelines for Drinking-water Quality in the form of the water safety plan (WSP) approach. Both initiatives recommended incorporating risk matrices within the risk assessment element of the WSP.

Since 2004, evidence from 97 different countries has validated and refined the use of a water safety planning framework to ensure safe drinking water is consistently provided to consumers (WHO & IWA 2017). WSPs are intended to provide a holistic risk management approach for water supply systems, grounded in proactive hazard identification and risk mitigation vs. simply complying with a suite of water quality parameters (Bartram et al. 2009; WHO 2012; WHO & IWA 2017). The WSP methodology relies on risk matrices and descriptions to categorize the likelihood of hazardous events in a water system, an analysis technique chosen for its usefulness in data-sparse situations. For these purposes, the following definitions (WHO & IWA 2017) specifically apply to health risks:

• ‘a hazard is a biological, chemical, physical, or radiological agent that has the potential to cause harm;

• a hazardous event is an incident or situation that can lead to the presence of a hazard (what can happen and how);

• risk is the likelihood of identified hazards causing harm in exposed populations in a specified time frame, including the magnitude of that harm and/or the consequences’.

Similar relationships exist between hazards, hazardous events, and risks for other types of hazardous events, all of which are the focus of applying a risk matrix. Various hazardous events are judged by personnel familiar with a drinking water system by estimating their specific likelihood and magnitude of their consequences and having the risk matrix assign a quantitative score to the resulting risk. The body of literature surrounding WSPs focuses primarily on benefits and challenges associated with the framework as a whole; in contrast, little review of the risk assessment techniques employed in WSPs has been conducted. In the remainder of this introduction, we present benefits and drawbacks of the risk matrix method to frame our subsequent analysis.

A primary rationale for introducing WSPs was to achieve a more comprehensive and preventive approach to ensuring safe drinking water. Implementation of WSPs aimed to move beyond a narrow focus of drinking water providers on compliance with endpoint testing for numerical guideline values (Gunnarsdottir et al. 2012). The WHO is one of the few organizations to define safe drinking water, something that is not defined in the U.S. Safe Drinking Water Act, for example. WHO (2022) states: ‘Safe drinking-water, as defined by the Guidelines, does not represent any significant risk to health over a lifetime of consumption, including different sensitivities that may occur between life stages’. Although this definition appears clear, consistently achieving the stated intent demands more than simply achieving endpoint compliance with numerical guideline values. Most notably, ensuring safety for microbial pathogens, which most clearly cause human illness from drinking water exposure, is substantially more complex. The WHO guideline approach for specific pathogens relies primarily on preventively ensuring achievement of performance standards (e.g., log removal targets) that are inherently more complex to understand and challenging to achieve than simple retrospective endpoint compliance testing against numerical guideline values for chemical contaminants.

A focus on preventive risk management measures is core to a risk-based approach. To understand and implement risk-based approaches requires a common understanding of what is meant by the risk that is to be managed. WHO (2022) defines risk as: ‘the likelihood of identified hazards causing harm in exposed populations in a specified time frame, including the magnitude of that harm and/or the consequences’. This definition demonstrates that risk is complex, multi-dimensional, and includes all the uncontroversial elements of a more expansive, fundamental definition of risk in relation to safe drinking water (Hrudey et al. 2011). The inevitable complexity and multi-dimensional character of risk underlies the challenges faced of providing WSPs with a practical, viable approach to risk assessment.

Risk matrices were selected for use in WSPs for their utility at prioritizing the level of risk associated with a specific hazard in comparatively data-scarce water systems (Bartram et al. 2009; WHO 2012). Small water systems do not necessarily have the data required to complete in-depth estimations of the likelihood of a hazardous event occurring, nor do they have the capacity to categorize all the potential impacts of a hazard (human health, environmental, cost, etc.). As a result, risk matrices were chosen to provide a simplistic, semi-quantitative method for identifying risks in water systems, recognizing their imperfections but preferring risk matrices to overlooking risks altogether. Risk matrices in WSPs consist of a series of likelihood and consequence/impact descriptors, most commonly having five different levels and scores, which are then multiplied together to generate risk scores. Risk scores are then assigned levels based on a specific boundary criterion.

How a risk matrix is constructed is critical to its effective, rational performance as a risk assessment tool. Risk matrices from one industry or organization are often transplanted into water system risk assessments without properly adapting the matrix to the specific public health impacts that need to be considered to protect public health. As a result, many water systems have risk matrices that attempt to evaluate financial, operational, management, and environmental risks all within the same framework without consideration for the industry it is intended to serve – one capable of harming the public's health. If the risk matrix, for example, predominantly underestimates the risk of a hazard, then a water system is more likely to devote fewer resources to monitoring and mitigating a risk that may have a large impact on the system. This example demonstrates two key flaws of risk matrices: suboptimal resource allocation and poor resolution (Cox 2008). Cox's study of risk matrix construction revealed that risk matrices do not necessarily support an ideal allocation of resources to mitigate risk if the risk matrix is not constructed with adequate resolution of risk estimates. Many risk matrices suffer from ambiguous inputs and outputs: likelihood terms are defined by words such as ‘unlikely’ or ‘likely’ which represent an extremely wide range of numerical probabilities that are often inadequately defined. Impact terms are often subjective to a given hazardous event, particularly when the hazard itself may have uncertain consequences. Consequently, the outputs from a risk matrix can be spurious at best, providing only a subjective allocation of risk (Cox 2008). Risk matrices rely on a user to understand and properly categorize risk (Cox 2008), a task which humans generally do not accurately perform because of their own subjective bias (Tyszka & Sawicki 2011; Idzikowska et al. 2017).

‘Poor resolution’ is a term which refers to the situation when a risk matrix generates the same risk score for quantitatively different risk events (Cox 2008). For example, a low probability, high impact event may have a risk score of 10 based on a likelihood score of 2 and an impact score of 5. However, a highly probable event with a likelihood score of 5, but a relatively small impact score of 2 will also have the same risk score of 10. While the risk score is the same, these events are quantitatively different. The first scenario (low probability, high impact) could feasibly represent a groundwater contamination event by Campylobacter jejuni, an event that is unlikely, yet can have devastating human health impacts. The second scenario (high probability, low impact) could feasibly represent the failure of an old, but redundant pump, an event that is likely to occur because of the pump age but will have a minimal impact because there are redundant pumps available. In the water industry, the impacts of a water system failure are both monetary and human health based, making the improper characterization of risk events potentially life-threatening to the consumers in the water system.

This study reviews how risk matrices are currently reported as being constructed for the risk assessment component of a WSP. We reviewed current WSP template documentation to locate risk matrices and compared the risk matrices to the statistical validity criteria set out by Cox (2008) to determine whether each risk matrix satisfies those criteria. In addition, we reviewed the descriptions of likelihood and consequence components of risk matrices from each jurisdiction and compared these definitions. This secondary analysis allowed us to evaluate how risk has been defined in different jurisdictions and whether there are consistent methods for determining likelihood and impact of events.

A comprehensive review of both literature and WSP templates was conducted to evaluate the use of risk matrices in WSPs globally. An initial review of literature using the key words ‘water safety plan’ AND ‘risk matrix’ returned few results in major databases such as Web of Science, Wiley Online Library, and ScienceDirect; most peer-reviewed literature of WSPs focuses on implementation and briefly mentions the risk matrix process. Therefore, a search of the WSPortal website was conducted, as this website contains templates, supporting documentation, and case studies of WSPs globally. A review of the available WSP templates and documentation on WSPortal (as of 2022) was conducted to identify different risk used in WSPs. Additional literature pertaining to the use and construction of risk matrices was included in this review to provide context when evaluating the risk matrices located. We identified 12 WSP templates with adequate information to complete a thorough review of the risk matrix structures.

We reviewed the risk matrix structure in detail, looking for the following items: (1) the descriptions and definitions of likelihood, (2) the descriptions and definitions of consequence, (3) the scoring paradigm used in the risk matrix, and (4) the boundary criteria for each risk level associated with a risk score. We then summarized the likelihood and consequence information into tables, including the scoring paradigm, the definition of each likelihood or consequence level, the description of each likelihood or consequence level, the scores associated with each level, and any other important information of note.

To determine whether the risk matrices used in WSP templates are properly constructed to assess risk, statistical validity criteria for risk matrices were compiled from Cox (2008). The author uses mathematical and logical principles to demonstrate the drawbacks to the risk matrix as a method of evaluating risk and presents the following risk matrix criteria in their work to aid in the identification of strong risk matrices:

• (1)

  Weak consistency – risks with a higher quantitative value must be represented with higher risk levels in a risk matrix such that every high level risk is quantitatively scored greater than lower risk.

• • a.

    Argument 1 – No low-risk level cell (green cells) should share an edge with a high-risk level cell (red cells).

  • b.

    Argument 2 – If you have at least two risk levels in a risk matrix (with low risk in the upper left and high risk in the lower right) and the axes increase in magnitude, then no high-risk level cell (red cell) is in the left column or top row of the risk matrix. If the risk matrix is drawn with low risk in the lower left and high risk in the upper right and the risk scores increase, then no high-risk level cell (red cell) is in the left column or bottom row of the risk matrix.

• (2)

  Betweenness – A matrix must be at least a 3 × 3 matrix where every negatively sloped line segment from a low-risk level cell in the top left to a high-risk level cell in the bottom right passes through at least one intermediate cell.

• (3)

  Consistent color coding – No risk level or numeric value for any risk level may intersect the numeric values from another risk level. In other words, a value assigned to low risk must be no greater than the lowest numeric value of intermediate risk which in turn may not be higher than the lowest numeric value of high risk. Supplementary Figure S1 presents a generic risk matrix with consistent coloring in Panel (a) and a generic risk matrix with inconsistent coloring in Panel (b) for illustrative purposes.

The criteria presented above are utilized in this study to evaluate the risk matrices available in WSP templates for their application of these three principles. The criteria were either evaluated as meeting the risk matrix criteria (in green), not satisfying the risk matrix criteria (in red), or indeterminate (in yellow). Indeterminate is here used for risk matrices where it is not clear if the risk matrix does or does not meet the criteria or not enough information is available to make an evaluation. In this study, the term ‘conform’ or ‘conformance’ is used to signify that a risk matrix complies with the criteria compiled by Cox.

Cox (2008) provides a specific critique of risk matrices by arguing the same risk score can be obtained for ‘qualitatively different risks’. For example, if both likelihood and impact are scored on a scale of 1–5, a likelihood score of 1 generally indicates an event which is rare or unlikely to occur in a risk matrix and an impact score of 5 is generally considered a severe or high impact event, yielding a risk score of 5. However, an event that is almost certain with a likelihood score of 5 and a minimal impact of 1 will also generate a risk score of 5. These two scenarios are qualitatively different in terms of their respective importance for drinking water safety. Yet a risk matrix yields the same risk score which can lead to misinterpretation of results and subsequent poor allocation of available resources to mitigate the risk (Cox 2008). Furthermore, the descriptions of likelihood score 1 in one jurisdiction (‘unlikely to occur once in 10 years’) may also be qualitatively different from another jurisdiction (‘Never occurs in this water system’). As a result, even if risk scores have the same numeric value across different jurisdictions, the same risk score can result in a qualitatively different description of the severity of the risk in the system.

As a component of our analysis, we considered the 2016 waterborne illness outbreak in Havelock North, New Zealand. The hazardous event we considered was the occurrence of a waterborne disease outbreak in the water system. The risk of this event was described in a 2008 WSP as ‘unlikely’ (likelihood) and ‘moderate’ (impact) (Government of New Zealand 2017; Graham et al. 2023). Ten of the 12 risk matrices identified in our critical review have descriptions that match these definitions. Using each risk matrix, we evaluated the risk score and level that would have resulted using these definitions (Scenario 1). We then also evaluated what the risk levels would have been if the impact was changed to ‘catastrophic’ (Scenario 2). We compared the description of risk in each scenario to determine which risk matrices would have adequately identified the high risk of waterborne illness that was present in Havelock North. This presents a real-world scenario of how risk matrix construction has important implications for the safety of drinking water by showing how improper risk matrix construction can lead to unidentified major risks in a water system.

Likelihood and impact definitions and descriptions from 12 different WSP templates were identified through our review process. Table 1 presents the likelihood score, definition, and consequence score, definition, and description.

The most common scoring regime for likelihood is a 1–5 score for 5 likelihood levels defined as ‘rare’, ‘unlikely’, ‘moderate’, ‘likely’, and ‘almost certain’. However, the definition of each of these levels varies across jurisdictions. For example, ‘rare’ in the WHO WSP manual is defined as ‘once every 5 years’, whereas in Malaysia, ‘rare’ is defined as ‘once every decade’ and ‘very uncommon event, unlikely to occur’ from the Pacific Islands WSP template. Although a scale of 1–5 is most common, an exponential scale based on powers of 2ⁿ is used in Alberta. The TECHNEAU manual does not use numerical values and is represented by NA in Table 1. South Africa's template is the only template to use decimals to score likelihood which plays a role in how the risk matrix is constructed as discussed in subsequent sections.

In Table 1, the majority of the likelihood descriptions refer to a defined timeframe during which a hazard may occur. Examples include: ‘every 5 years’ or ‘every week’. It is most common to see a likelihood definition of ‘almost certain’ corresponding to a daily or weekly occurrence. However, rare events differ according to jurisdiction. For example, in Iceland, a ‘rare’ likelihood definition is described as an event that occurs once every 100 years. In contrast, in the WSP manual, the TECHNEAU manual, Alberta and South Africa templates, ‘rare’ is defined as every 5 years. Different descriptions of a rare event are particularly problematic when interpreting risk as it can lead to an underestimation of low probability, high impact events such as drinking water outbreaks.

In addition, risk scores show a fundamental flaw in using a 1–5 scale to capture probability descriptions such as ‘once a year’ and ‘once a week’. A probability of once every 5 years corresponds to one event every 1,825 days while a probability of once every year is one event every 365 days and a probability of once a week is one event every 7 days. A risk scoring scale of 1–5 has equal intervals between each risk score, but the actual probability values corresponding to the descriptions have much larger intervals and intervals that are not the same.

The risk descriptors provided in Table 1 for New Zealand are those recommended by the New Zealand Ministry of Health (NZMOH 2014) when it updated terminology to WSPs from the original terminology (NZMOH 2001) that referred to Public Health Risk Management Plans. These descriptors were in place before the 2016 Havelock North fatal outbreak. Subsequently, guidance for WSPs from the New Zealand Ministry of Health (NZMOH 2019) did not prescribe these descriptors, presenting them as examples. Currently posted guidance from the new national water regulator (Taumata Arowai 2023) refers to the first edition of the WHO WSP guidance manual (Bartram et al. 2009).

It is important to note in Table 1 that five of the templates define likelihood in terms of events that have already occurred in the water system. While using historical data to predict future conditions is important to understand likelihood, the probability of a future event does not depend on past events if we treat each event as an independent occurrence. For example, in Ireland, ‘almost certain’ is described as ‘Has not happened in the past and is highly improbable that it will happen in the future’. This description implies that because an event has not occurred in the past it is not likely to occur in the future; however, previous studies (Gluckman & Bardsley 2021) have shown this is not true for low probability, high impact events. An example of a low probability, high impact event is a natural disaster, which cannot always be predicted based on past data, but has the potential to have a large impact on a water system. Walker (2023) has described, in some detail from extensive experience, how reliance on likelihood of equipment failure estimates based primarily on estimates from asset managers can lead to underestimating the risk of catastrophic health consequences arising from latent risks resulting from the absence of adequate barriers to microbial contamination. Five of the templates above (Ireland, Alberta, the Pacific Islands, Bhutan, and Nepal) construct likelihood descriptions using past probabilities of occurrence which may lead to the underestimation of the probability of occurrence of future events.

Table 1 shows that consequence is also quantified in different ways across jurisdictions. In some locations such as the WSP Manual, Ireland, Alberta, Malaysia, Bhutan, and Nepal, impact can be tied to compliance or regulatory impacts which are jurisdiction specific. In Alberta and Nepal, impact is also described in terms of the duration of a service interruption, tying the impact to water quantity. In both Bhutan and Nepal, impact is also described in terms of the number of customer complaints about water quality. The WSP manual, Ireland, TECHNEAU, and South Africa templates described at least one impact in terms of aesthetic concerns with the water quality in the water system.

In all the consequence descriptions available, six of the impact descriptions are tied to health impacts or the consumption of potentially contaminated water by customers. However, in some of the templates, water quality concerns are divided between aesthetic water concerns and microbial water concerns, which in practice do not have equivalent impacts on human health. Aesthetic impacts largely do not have a physical health impact on customers; they may reduce consumer confidence and change behavior – i.e., expending a limited household budget on bottled water. In contrast, even small microbial contamination events have the potential to have a large adverse impact on customers. Risk matrices that treated both health and aesthetic concerns as ‘water quality’ are effectively equating the impact magnitude of these different events.

In locations such as Ireland, Bhutan, and Nepal, we also observe multiple different types of impacts associated with the same consequence score. Grouping impacts has the benefit of covering many possible scenarios at once. However, the descriptions do not prioritize the importance of one class of impact over the other (e.g., disease outbreak vs. service interruption); therefore, systems experiencing two different impacts at the same time may struggle to categorize the importance of one impact in relation to another, potentially leading to underestimation or overestimation of risk in the system. One recommendation to improve these templates is to consider each major class of impact separately, in a different risk matrix to avoid underestimating impacts of specific hazards.

Exponential impact scales are appealing because these scales generate a wider range of risk scores that can be easily differentiated in a risk matrix. For example, in Alberta, an exponential scale of 1–16 for both probability and impact results in risk scores that range from 1 to 256. However, the catastrophic impact of the waterborne disease outbreak in Walkerton, Ontario caused 7 deaths, over 2,000 cases of illness and cost more than $64 million in economic impacts (Hrudey & Hrudey 2014) but is only 16 times worse than an insignificant impact evaluated using the Alberta risk matrix. Scoring scales in risk matrices are used to simplify the analysis of risk for hazardous events and provide recommendations for prioritizing risk; however, the example from Walkerton demonstrates that even an exponential scale for impact needs to cover a sufficient numerical range to represent the full range of impacts that can occur in water systems.

Our evaluation of likelihood and impact definitions, scores, and descriptions showed that there is no consensus across WSP templates as to how to best define both the probability of a hazard and the impact of that hazard. While system specificity is necessary and important to identifying risks properly, we recommend future WSP guidance include methodologies that teach a WSP team how to construct a risk matrix that avoids the flaws we have noted. As seen with the analysis of the Havelock North waterborne outbreak (Tables 2 and 3), public health implications are only included in overall risk descriptions for the highest impact definitions, with impacts being inconsistently defined in terms of aesthetic vs. water quality impacts vs. regulatory compliance breaches within the same risk matrix. If a water system is starting the WSP process, many stakeholders will select the risk matrix present in the WSP manual or search other jurisdictions to find a risk matrix that may work in their system. As we have shown, there are at least 12 different WSP templates or water quality guidelines with different risk matrices, challenging a WSP team to choose an effective method most suited to their water system.

Table 2

Evaluating the Havelock North water system description of risk based on ‘unlikely’ likelihood and ‘moderate’ impact (Scenario 1)

In the risk description, terms related to public health impacts are highlighted in red.

View Large

Table 3

Evaluating the Havelock North water system description of risk based on ‘unlikely’ likelihood and ‘catastrophic’ impact

In the risk description, terms related to public health impacts are highlighted in red.

View Large

Those completing a WSP are not likely to be risk assessment experts; therefore, we recommend future WSP documentation should provide an activity or section in the WSP manual dedicated to examples or risk matrices and guidance on selection of likelihood and impact factors. As a result, defining the likelihood levels in a risk matrix is extremely important, particularly for low probability, high impact events. Previous studies have shown that laypeople have difficulty comprehending low probabilities of an event (Tyszka & Sawicki 2011; Idzikowska et al. 2017); subsequently, we highlight the importance of water stakeholders coming to a consensus on the definitions of likelihood descriptors such as ‘unlikely’ or ‘rare’.

Furthermore, we recommend public health impacts must always be one of the impacts that a jurisdiction includes within their risk matrix construction. As seen in Tables 2 and 3, public health was always considered in the description of a ‘catastrophic’ impact but was only included in the description of impact in four of eight risk matrices for a ‘moderate’ impact. Impact descriptions at a minimum need to consistently include public health related terminology. The purpose of a WSP is ‘to consistently ensure the safety and acceptability of a drinking water supply’ (Bartram et al. 2009) and is based on a holistic risk-based method aimed at moving beyond endpoint monitoring to ensure the protection of public health. Risk matrices constructed for use in WSPs should reflect this purpose by clearly defining impact in terms of public health implications. Supplementary information provides readers with suggestions for how risk matrices may be improved.

The risk matrices evaluated in this study most often fail to meet the risk criteria for betweenness and weak consistency (Argument 2). This is often the result of scoring systems that do not provide a wide enough range of values to adequately delineate between definitions of risk levels (low, moderate, etc.). Argument 2 states that if the risk matrix is drawn with low risk in the lower left and high risk in the upper right and the risk scores increase, then no high-risk level cell (red cell) is in the left column or bottom row of the risk matrix. For example, in the Alberta and Pacific Island Matrices, the bottom row of the risk matrix has a moderate risk level resulting from how the low-risk score is defined. Consequently, we evaluated both the Alberta and Pacific Islands risk matrices as ‘Indeterminate’ for the Argument 2 criteria.

The betweenness risk matrix criterion in WSP risk matrices was only met in half of the matrices evaluated. This is a result of (1) too few risk scores in the risk matrix corresponding to moderate level risk or (2) a risk matrix that is a 3 × 3 matrix. For example, in the Nepal risk matrix, a score of 3 or 4 will result in a moderate level risk. If a line is drawn from the top left corner to the bottom left corner in the risk matrix, the line should pass from low to moderate to high risk without missing the moderate level risk. Figure 3 shows that in the current Nepal risk matrix (Panel (a)), a line can be drawn that satisfies betweenness from a risk score of one through a risk score of 4 to a risk score of 9. However, if a line is drawn from a risk score of 2, it will not pass through a moderate level risk before intersecting a risk score of 6 which corresponds to a high level risk in the risk matrix. To solve this issue, the boundary conditions for low risk could be set such that low risk is a score of 1, moderate risk is scored 2–4, and high risk is 6–9 (shown in Panel (b)). While betweenness is now satisfied in Panel (b), this risk matrix now may be more likely to overestimate risk in a system since there are five moderate risk cells compared to the original three in Panel (a). Furthermore, if low risk was defined as a score of 1–3, moderate risk was scored as 4–6 and high risk is >6 (Panel (c)), betweenness would again be satisfied but the risk matrix would have the tendency to underestimate risk as there are now five cells corresponding to low risk instead of the original three. Because the Nepal risk matrix is a 3 × 3 matrix, there is little flexibility to both meet the betweenness criteria and still construct a risk matrix that does not over or underestimate the risks in a water system. This result indicates, as Argument 2 did, that the delineation between each level of risk is critical to risk matrix construction and ultimately utility. Furthermore, the need to have risk matrices larger than a 3 × 3 matrix to allow for more flexibility in risk matrix construction is indicated. In any case, it should be clear that the quantitative construction of a risk matrix that seeks to satisfy the Cox criteria may be challenging. Users need to explore realistic scenarios to satisfy themselves that the risk ranking provided by their adopted system satisfies what would be judged reasonable in hindsight if the failure comes to pass. The next section explores these challenges.

For Havelock North, New Zealand, prior to a waterborne disease outbreak, the likelihood of an outbreak was defined as ‘unlikely’ and the impact was described as ‘moderate’. Table 2 presents the results of using these likelihood and impact definitions to evaluate the risk of a waterborne disease outbreak (Scenario 1). Annual events were assigned an ‘annual’ or ‘once every year’ likelihood score for consistency across matrices to mimic the best case scenario for each risk matrix. No risk matrix indicated that waterborne disease outbreak was a high risk for this community, although in actuality four people died and at least 5,500 people fell ill as a result of Campylobacter presence in the water supply (Government of New Zealand 2017; Graham et al. 2023). Of course, the ratings of likelihood as unlikely and impact as moderate for an outbreak in Havelock North were not accurate. In addition, only 3 of the 10 risk matrices result in a risk description that includes words related to public health implications (shown in red text in Table 2). In the WHO, Alberta, South Africa, Nepal, Australia, New Zealand, and Malaysia results, there is no consideration of public health impact even though the hazardous event being evaluated is directly related to public health. Furthermore, half of the risk matrices indicate that an ‘unlikely’ event with ‘moderate’ impact is considered low risk, with the Bhutan risk matrix alone acknowledging public health, but indicating that there is no health impact expected. Table 2 shows that even when the likelihood and impact definitions are the same, the score associated with that definition is different, the description of likelihood and impact is different and as a result risk is subjective.

In Scenario 2, we changed the impact definition from ‘moderate’ to ‘catastrophic’ to reflect the actual impact that was experienced in Havelock North (Table 3). The first noticeable difference is the presence of a health-based risk description in 8 of the 10 risk matrix results. Impacts are inconsistently defined within risk matrices. For example, in South Africa, a moderate impact in Scenario 1 is related to aesthetic water quality impacts, but a catastrophic impact in Scenario 2 is ‘death expected from exposure’. Even with an impact of ‘death expected from exposure’, the risk level obtained using the South Africa risk matrix is only moderate, not high risk. Only three risk matrices would have identified the hazardous event as high or very high risk: the WHO Manual, the Australia Drinking Water Guidelines Manual, and the New Zealand risk matrix. Only two risk matrices, using different combinations of likelihood and impact scores were able to adequately characterize the severity of the hazardous event in Havelock North, but it is notable that in each of these risk matrices (Australia and New Zealand) that the description of risk does not include language about public health impacts. Assigning ‘once every year’ as a best case scenario (lower risk score) interestingly generates a risk level of high in the WHO matrix; however, it is important to note that an event that has occurred once in a year can also be scored as ‘once every 5 years’ in many of the matrices shown in Table 1 if the event does not repeat in a subsequent year, resulting in all risk levels being moderate in Table 3. Therefore, even if the risk of the event had been conservatively estimated with any risk matrix, there would still have been the potential to misrepresent the level of risk is a system.

In Table 3, there is more consistency between descriptions of risk in Scenario 2, with more risk descriptions acknowledging public health impacts. However, of the four risk matrices that scored risk as 10 (WHO, Ireland, Pacific Islands, and Malaysia), the risk descriptions disagree about the probability or frequency of the event occurring despite all using ‘unlikely’ as the probability definition. ‘Unlikely’ in Ireland means ‘the hazard is possible under specific circumstances’ which is flexible enough to encompass a variety of events, but too vague to be useful to make a decision about risk mitigation steps. The Malaysia risk matrix indicates ‘unlikely’ is an event that occurs ‘yearly’ while the Pacific Islands define ‘unlikely’ as an ‘event that may not occur’. The meaning of ‘unlikely’ as a descriptor for likelihood may present a problem by being more value-laden than something like ‘rare’ or ‘very infrequent’. Over-confidence in the response capabilities by the water provider could lead to an understanding that rare, high consequence events are ‘unlikely’ because they would be prevented, rather than correctly assigning the frequency of the specified hazardous event occurring.

Risk is a concept historically not well understood by humans (Kunreuther et al. 2001) and human error in completing risk assessments is only exacerbated when the risk assessment procedure is not easy to understand. Water systems failures are most often a chain of individual failures that lead to a catastrophic outcome, instead of one specific hazardous event being high risk (Gluckman & Bardsley 2021). While our results show that proper use of a risk matrix should have helped to identify an individual hazardous event, there are many additional factors that contributed to the outbreak in Havelock North (Graham et al. 2023). Risk matrices are only one tool that can be used to identify hazards and to characterize and judge the relative importance of risks in a system. Ultimately, WSPs and the risk matrix component are intended to ensure that operational staff fully understand their systems and are able to apply informed judgment to ensure that important risks are effectively managed. Our study shows that risk matrix construction needs improvement and part of the improvement will involve working in teams of stakeholders (part of the WSP process) to collaboratively construct and understand a risk matrix to improve its utility to the risk assessment process in the water industry. Quantitative results from a risk matrix should never overrule sound knowledge about ensuring drinking water safety including that based on relevant experience reported by others (Hrudey & Hrudey 2004, 2014).

Using guidance from Cox (2008) and the likelihood and impact descriptions from each WSP template, we examine how WSP risk matrix construction could be improved. Detailed suggestions to adjust risk matrix construction to satisfy the Cox criteria are provided in the Supplementary Information. However, the primary purpose of a risk matrix for WSPs, recognizing the risk matrices have been widely adopted across many, diverse risk management applications, some with demonstrably recognized challenges (Vatanpour et al. 2015), must be to encourage and facilitate a mature and informed discussion among operational personnel about the comparative importance of identified risks. Likewise, as Cox (2008) originally noted: ‘risk matrices should be used with caution, and only with careful explanations of embedded judgments’. In particular, there is arguably no type of risk more critical for a drinking water utility to prevent than causing consumers to become ill from consuming contaminated water supplied. Consequently, if risk matrices are to be used effectively to promote meaningful risk priority discussions among operational personnel, the matrices need to avoid suggesting or implying that the risk of allowing a drinking water outbreak is anything other than catastrophic. A simple measure in this regard will be to prepare a separate risk matrix for judging outcomes substantially affecting public health from issues affecting aesthetic or economic factors.

Our results indicate that risk matrices in WSPs only conform to some of the statistical criteria set out by Cox (2008). While the challenges presented do not warrant risk matrices being abandoned, risk matrices do need to be restructured in WSPs to provide better guidance to water systems for evaluating risks to such systems. We also note that the most recent WSP guidance from WHO (Jackson et al. 2023) has been much less prescriptive about a single structure for risk matrices, choosing to present them as illustrative examples. Likewise, the Australian Drinking Water Guidelines (NHMRC 2023) note that there is inherent uncertainty and subjectivity to any risk matrix that needs to be accounted for in risk assessments.

Risk matrices have been promoted for use in water safety planning to prioritize improvements to water systems infrastructure and management. However, no published studies have examined whether risk matrices used in this global water management methodology are constructed in accordance with underlying mathematical theory. In this study, we examined risk matrices used in WSPs globally; our results indicate that 11 of the 12 international risk matrices we evaluated did not conform to at least one of the criteria set out by Cox (2008). In addition, we discovered that likelihood and impact definitions vary considerably across jurisdictions partly because of the system-specific nature of the WSP methodology. To improve risk matrix construction, risk practitioners should better define risk level boundary criteria to eliminate risk matrices that do not conform to the Cox (2008) criteria. Furthermore, risk matrices should consider one specific impact category in each individual risk matrix as opposed to describing impact in several ways to avoid inaccurate risk scoring. Better definitions of rare events, potentially informed by water system monitoring data, provided they do not lead to the flaws exposed by Walker (2023) could also serve to strengthen the accuracy of risk matrix predictions. Catastrophic failures, such as drinking water disease outbreaks, need to be better distinguished from other less serious matters. Finally, to ensure risk matrices are an accurate reflection of relevant water system risk, we recommend including a review of risk matrix construction, including likelihood and impact definition in the WSP improvement cycle.

The authors would like to thank Tony Cox, Associate Professor from the University of Colorado-Denver for explaining his risk matrix paper and providing important recommendations for our consideration in the development of the methodology for this paper.

All relevant data are included in the paper or its Supplementary Information.

The authors declare there is no conflict.