Assessing clarity of message communication for mandated USEPA drinking water quality reports

There are over 52,000 community water systems (CWSs), also known as drinking water utilities, across the USA that collectively serve over 300 million people (United States Environmental Protection Agency (USEPA) 2012c). In 1996, Congress amended the Safe Drinking Water Act, requiring any CWS with at least 15 service connections or serving more than 25 people to provide annual water quality reports to their consumers (USEPA 1999). As a result, in 1998 the consumer confidence report (CCR) rule was enacted to require CWSs to distribute information regarding water quality by 1 July of every year (USEPA 1999). Even before the initial implementation of CCRs, nationwide studies found that consumers would read, and wanted, more information about their tap water (Roper Starch Worldwide 1999; Benson et al. 2002; Meyer-Emerick 2004).

According to the USEPA, the purpose of the CCR rule is to: ‘Improve public health protection by providing educational material to allow consumers to make educated decisions regarding any potential health risks pertaining to the quality, treatment, and management of their drinking water supply’ (USEPA 2009). The USEPA mandates that CCRs present information for four major areas: (1) source water type; (2) water quality; (3) compliance; and (4) educational information. In an effort to standardize communications, many CWSs use the CCR-iWriter software maintained by USEPA (2013) to construct their CCR. CCR-iWriter provides the required and essential contaminant table where CWSs input their water quality data (Table 1). The CCR-iWriter also provides standardized scripts for several categories, including the Lead and Copper Rule, health information for immuno-compromised individuals, and water conservation strategies.

Since the CCR is considered a public health communication (USEPA 2009; Roy et al. 2015), its information and messages should be readily understood by all consumers. Unfortunately, to date, criteria have not been established, and CCRs have not been systematically evaluated to ensure that the CCR messages are clearly communicated in a manner that consumers can understand. In 2001 and 2013, national consumer water quality surveys discovered that 86% of adults (Means et al. 2002) and 71% of adults (Carpenter & Roberson 2013), respectively, surveyed had concerns regarding their drinking water quality. Reports indicated that only 17% of concerned consumers had even received or read their CWS's CCR (Means et al. 2002). When CCRs were first distributed, many utilities were concerned by the lack of inquiries from consumers and suggested that one reason could be that consumers were unable to read and comprehend the information in the CCR (Benson et al. 2002). Over a decade after implementation of the CCR rule, the USEPA and stakeholders reviewed it and identified the need to improve effectiveness of communicating information to consumers, with the number one area of concern being CCR understandability (USEPA 2012a).

Consumers can only make informed decisions when scientific information is presented in a way that is objective, understandable, and interesting (Huntzicker 1980). It is not that the water issues are difficult to grasp, but it may be the way that CWSs communicate them that makes them difficult to grasp (Huntzicker 1980; Roy et al. 2015). The use of scientific jargon or complex technical terms complicates the ability of consumers to make informed decisions on issues related to increased personal risk (e.g., water/energy/air quality; toxic chemicals) (Huntzicker 1980). The majority of consumers have little previous exposure to the jargon or technological concepts (Dietrich et al. 2014) so they cannot fully understand environmental issues that influence personal risk (de Vore 1992; Mihelcic et al. 2003).

Further complicating the issue, it is estimated that over one-third of American adults experience low health literacy, which is defined as the extent to which an individual has the capacity to obtain, process, and understand basic information needed to make appropriate health decisions (Paasche-Orlow et al. 2005; Davis et al. 2006; Vernon et al. 2007). Low health literacy of Americans is a concern as readability, or comprehension difficulty, of CCRs was determined to be at the 11th–14th grade level, which greatly exceeds the recommended 6th–7th grade level for public health communications (National Institutes of Health [NIH] 2013; Roy et al. 2015). While the readability of CCRs was assessed to be beyond the reading level of many Americans, their ability to clearly communicate messages with clarity has not been assessed.

Concerns about understandability of health communications for diseases and medications led the Centers for Disease Control and Prevention (CDC) to study the importance of message communication to consumers and develop the Clear Communication Index (Index) as a research based tool for preparing effective and clear health communication materials for consumers (CDC 2013). The goal of the Index is to improve the clarity, or the ease of understanding messages, of consumer communication products and make materials easier to understand by the intended reader (CDC 2013). The Index evaluates written documents on seven indices: (1) Main Message/Call to Action; (2) Language; (3) Information Design; (4) State of the Science; (5) Behavioral Recommendations; (6) Numbers; and (7) Risk. Explanations for each category are provided in Table 2. A study by Baur & Prue (2014) evaluated the benefits of redesigned health material using the Index criteria and found that the redesigned version rated higher than the original document. They concluded that implementing the Index tool improved main message communication and increased understanding of words and numbers (Baur & Prue 2014).

Since CCRs are communicating messages about health and risk, such as water contaminants and quality, the Index tool is appropriate to evaluate effectiveness of CCRs. This research assessed the clarity of message communication of CCRs for a national representation of different US regions serving different size populations. The specific objectives were to: (1) apply the CDC's Clear Communication Index (Index) to evaluate the overall clarity and format of CCRs; (2) evaluate the CCR according to the seven indices of the Index; and (3) assess the ability of CCRs to communicate scientific information and risk.

Thirty CCRs were obtained that represented CWSs from 29 different US states. CCRs were a stratified random sample from each of the 10 USEPA regions; there was one CCR for each of three population size categories per region. These were the same CCRs used in previous research (Roy et al. 2015) and were obtained in pdf through the USEPA's and CWSs' web pages from years 2011–2013 (USEPA 2014c, 2014d). Size categories for CWSs were based on the population served: medium (3,301–10,000); large (10,001–100,000); and very large (100,000+). These size categories represent 70% of the US population that is served by CWSs, or about 210 million consumers (USEPA 2012c).

The Index is a scientifically based tool that requires the users to be properly trained to effectively use it and minimize subjectivity. Two of four CDC-Index-trained researchers with expertise in health and/or environmental engineering evaluated every report independently. One environmental engineer rated all 30 CCRs; the other rater was either an environmental engineer or a health scientist. A Cohen–Kappa statistical test was used to determine inter-rater reliability.

The Index has four parts: (A) Main Message and Call to Action; (B) Behavioral Recommendations; (C) Numbers; and (D) Risk. These are further split into seven indices: (1) Main Message/Call to Action; (2) Language; (3) Information Design; (4) State of the Science; (5) Behavioral Recommendations; (6) Numbers; and (7) Risk, with a total of 20 scored questions (Table 2). The scores from each index were also tallied to obtain an overall score (out of 100%); the Index considers overall clarity scores = 100% ‘perfect’, scores ≥90% ‘passing’, and that any materials with scores <90% should be revised and improved until the Index score is ≥90% (CDC 2013). The authors also rated each CCR based on its ability to communicate a primary health message, i.e., the question, ‘Is the water safe to drink according to all state and federal standards and regulations?’ The researchers considered this topic the main health message of interest to consumers even though the USEPA CCR rule does not require that CCRs state explicitly whether water is safe or not.

According to the data shown in Table 3, the average number of pages was 6.2 ± 6.1 with no statistical difference among CWS sizes (one-way analysis of variance (ANOVA), p-value = 0.64), 50% had images, and 63% had color for the sampled CCRs. Very large CWSs were all printed in color and tended to have more pages and pictures than large and medium water utilities. The format difference among medium and large versus very large CWSs may be due to staff and financial capacities (Bishop 2003).

All CCRs provided required information, such as source water, water quality data, compliance information, and educational information. Water quality data were presented in tables that listed the contaminant, maximum contaminant level (MCL), maximum contaminant level goal (MCLG), average reported level, major source of contaminant in drinking water, and some did list health effects (Table 1). For the 30 CCRs reviewed, all appeared to use the CCR-iWriter software because the contaminant tables and educational text were the same as those provided by the CCR-iWriter. Educational information mainly pertained to conservation techniques, sources of contaminants, and health information. Although individual CCRs had different information and different styles, overall, there were similarities to easily recognize the document as a CCR.

A Cohen–Kappa statistical test indicated that the overall clarity scores were within statistical agreement among the four raters (p-value = 0.87). Unfortunately, all CCRs evaluated did not obtain passing Index scores because they failed to meet the recommended 90% overall scoring qualifying mark. The range for individual CCRs was 23 to 72%, with a mean score at 50 ± 14%. The overall clarity scores for medium, large, and very large size CWSs were at 49 ± 15%, 46 ± 18%, and 56 ± 9%, respectively, as shown in Table 3. A one-way ANOVA (p-value = 0.58) indicated that there was no significant difference for overall clarity ratings among the CWS size categories.

Only 63% of all CCRs sampled directly informed their consumers if the water was safe to drink according to state and federal standards and regulations. For medium, large, and very large sizes, the percentage of water utilities was 60%, 40%, and 90%, respectively. In reference to this question, a chi-square test found that the very large CWSs were statistically different from medium and large CWSs (p-value = 0.03). Most CCRs in the medium and large size did not mention any such statements and left the consumers to interpret the complicated jargon and technical contaminants table (Table 1).

The average scores for the seven key index areas were below passing throughout (Table 4). Analysis indicated that there was consistency across all CCRs as to why they were not scored as passing for each of the seven indices of the Index.

Behavioral Recommendations scored the overall highest at 77 ± 36% whereas the State of the Science scored the overall lowest at 3 ± 15%. A typical Behavioral Message was asking the consumers to conserve water, while an exemplar State of the Science statement included when CWSs explained what caused a contaminant spike. The Information Design index scored second highest, followed by Risk, Numbers, Main Message/Call to Action, and Language, decreasing in performance, respectively. Other exemplar examples include: using section headings for Information Design; providing detailed explanations of Risk associated with water quality for immuno-compromised individuals; using whole numbers to report MCLs; using visual clues like colors and boxes to highlight Main Message and Calls to Action; and clearly defining language that the lay consumer may not use or understand.

The only index that was statistically different (two-way ANOVA; p-value = 0.04) among CWS sizes was Behavioral Recommendations from very large utilities (92 ± 24%) performing better than medium and large utilities at 76 ± 38% and 62 ± 45%, respectively. The raters noted that most very large CWSs had pages or sections dedicated to Behavioral Recommendations for performing water conservation and maintaining premise plumbing infrastructure.

The primary purpose of this research was to determine the degree to which CCRs clearly communicated information on water quality to consumers. Based on the results of this study, coupled with previous research on readability (Roy et al. 2015), it is unlikely that a large proportion of consumers will understand the messages provided. The ratings of the CCRs examined within this study suggests that the reports fulfill regulatory mandates, but do not achieve the overall intent of the regulations – to improve population understanding of the degree to which local water is safe to drink. The CWSs are restricted in a sense by the requirement to use certain USEPA language in the CCR, although a CWS can add more information than the minimum required by the USEPA. The discussion that follows provides a summary of the study findings with related recommendations for CCR improvement. Each of the seven indices are addressed; Table 2 summarizes the intent of each index.

• (1) Main Message and Call to Action

The main message and sub-messages should follow the guidelines in the Index for delivering a clear message to consumers. Whether the water is safe to drink according to state and federal regulations should be clearly stated as the opening statement of the CCR. This suggests that the CCR format presents this section first. Most often, CCRs presented source water information first, and compliance with federal regulations last, although intermixing of the four areas was common.

The Main Message design should be at the top, beginning, or front of the CCR and emphasized with visual cues and graphics that are identifiable without effort from the reader (Baur & Prue 2014). The Call to Action must also be after the Main Message. The purpose of a Call to Action through the Index is ‘what you want people to do after receiving and understanding the main message’ (CDC 2013). Furthermore, ‘even if the purpose is to inform an audience, think about why they need this information, and use this insight to create a call to action’ (CDC 2013). Through Call to Actions, the Index encourages people to get involved, for example, clearly displaying phone numbers in the CCR for consumers to call for clarifications on their water quality. A Call to Action could also encourage consumers to change their behavior through using water conservation methods. Since CCRs are annual reports, their Call to Action should not be used for an emergency action like a boil water notice. If the CWS had any violation of a contaminant standard, the Call to Action could direct consumers to seek further details if interested, i.e., remediation steps taken or educational information about associated risks.

While CCRs provide information about water sources, treatment, and meeting regulatory limits, CWSs must realize that, when necessary, it is better to communicate the ‘bad news’ from the source itself rather than allowing it to get out another way (Berberich et al. 1998). The overall goals are educating and building confidence in the consumer, thus, CWSs must communicate all water quality information, including those which are negative or unpleasant (Berberich et al. 1998). Even if CWSs did not meet all water quality standards, they should report the findings because consumers need to know this information. CWSs can be successful in communicating information to their stakeholders through these 10 principles of authentic communication: communication must be truthful, fundamental, comprehensive, relevant, clear, consistent, accessible, timely, compassionate, and allow feedback (Bishop 2003).

• (2) Language

The language used in CCRs would be improved if active voice was used rather than the typical passive that is currently used. Active voice sentences are less wordy and more to the point than passive voice sentences and are a better way to communicate a direct message to your audience (Plain Language 2014).

The language should be words which are common or frequently used by consumers (CDC 2013). When it is unavoidable to use language that the consumer will not understand, the CWS must explain the terms in a lay-person language (Bishop 2003; CDC 2013). While the environmental engineer raters found the language familiar, but jargon, the health scientists raters had strong opinions about the unfamiliarity of several terms in the CCRs including: ‘cross-connections’, ‘greensand filters’, ‘sodium hydroxide’, ‘phosphoric acid’, ‘curb-stop’ and ‘MCLs’. These technical terms should be replaced or explained. A study that surveyed CWSs found that residential consumers were the hardest audience for CWSs to communicate with, followed by citizen's groups, the news media, business consumers, regulators, elected officials, and employees (Bishop 2003). Thus, CCRs have the potential to be an effective water quality communication tool for these groups through the use of active voice, day-to-day language, and defined terms (Bishop 2003).

• (3) Information Design

Most CWSs used bulleted or numbered lists as recommended by the Index; some of them, however, had more than the seven recommended items per list with no subheadings. Sometimes, lists were not used where they would have been appropriate. Additionally, the main message was not easily identified on the first page and/or in the first section of most CCRs. To effectively communicate the water quality message, it is imperative to make the main message easily identifiable to the consumer. Even though very large CWSs tended to have more pages, color, and pictures, the overall Index scores were not better than the scores for medium and large CWSs.

• (4) State of the Science

The State of the Science rated poorest out of all seven indices because CCRs did not provide authoritative sources for information on water quality or current science in toxicity, monitoring, regulations, and risk. Another failing of CCRs was not acknowledging uncertainty about data because science is continually evolving to produce new information (CDC 2013). To enhance the public's science literacy, CWSs need to demonstrate to consumers that findings and recommendations may change; by acknowledging those uncertainties, they can educate the public on the scientific process (CDC 2013). For example, while providing too much information can lead to negative effects (Owen et al. 1999), any contaminant detected above the MCL should be listed with the source of contaminant and potential health effects (Berberich et al. 1998). When a CCR reports that 99.5% of water samples met government standards, the remaining 0.5% samples may cause adverse judgments to be made due to exaggerated beliefs about the effects and magnitude of health risks associated with the failing 0.5% (Fischhoff et al. 1978; Meyer-Emerick 2004). The CWSs should use this opportunity to educate the consumers so they can recognize the minimal risks associated with the 0.5%, and ensure that the water is still safe to drink.

• (5) Behavioral Recommendations

Behavioral Recommendations ranked highest among all index scores. Many CCRs had recommendations such as water conservation, which is perfect for consumer education, but many CCRs lacked well written health-based recommendations. A poorly written example behavioral recommendation is the standardized script in the CCR-iWriter software describing risks for immuno-compromised people (USEPA 2013):

• (6) Numbers

The USEPA has a mandated table that CWSs must use to display numbers for contaminant concentrations and violations (Table 1) (USEPA 2012a). Contaminants are listed in unfamiliar units, like pCi/L, and will only be easily understood by technical people (USEPA 2012a). Also, numerical data intimidate many people, thus, numbers should be present in a way that the primary audience can understand, with whole numbers always preferred to decimal and fractions, and never requiring the audience to perform calculations (USEPA 2012a). The 30 CCRs did not use whole numbers as most water quality data were reported as decimals. Fortunately, no CWS required their consumers to do calculations.

Recently, the CCR rule retrospective review summary found that MCLs should be reported in numbers and units greater than or equal to 1.0, even though the regulation may be written differently (USEPA 2012a). For example, the MCL for benzene is 0.005 mg/L, but could be reported in the CCR as 5 ppb. This suggested change is also consistent with recommendations of environmental groups, which suggest to ‘not worry about units and just express information in whole numbers … if the standard is 20 and you have 2, that's a good message consumers will understand’ (Berberich et al. 1998).

• (7) Risk

Few CCRs stated the risk of how people may be affected by the water quality. Uncertainty in risk is a hard concept to express to consumers, thus, risk should be presented as probabilities in percentages (10% etc.) or mathematically equal proportions (1/10 etc.) (Boyle & Holtgrave 1989). It was found that risk is perceived as higher if presented in percentages versus proportions, but the proportions were considered higher when the denominator was 10 or higher (Boyle & Holtgrave 1989). Additionally, numbers alone are not enough, especially if they are small quantities or presented in unfamiliar units; people can fully understand risk and accept it only if they understand the physical processes that creates and regulates it (Fischhoff et al. 1978; Morgan et al. 1992). All CCRs should provide an explanation following any standard violation in order to alleviate concerns, such as public outcry and fear in consuming tap water. Such explanations should include what steps were taken to remediate the issues and potential risks to consumers when the violation occurred. Acknowledging violations and addressing unexpected water quality concerns in the CCR will promote public confidence.

Consumers use taste, odor, and appearance as indicators of safe drinking water (Dietrich & Burlingame 2015). It is human intuition to avoid products that are unpleasant due to risk perceptions that will shape their behaviors (Renn 1991; Jardine et al. 1999; Dietrich 2006; Doria 2010; Schade et al. 2015). Thus, CWSs should acknowledge and address unusual taste, odor, and appearance events even if it is only an aesthetic issue that does not violate regulated health-based drinking water standards. CWSs can invoke aesthetic guidelines, such as USEPA secondary maximum contaminant levels (USEPA 2012b), to determine when reporting an aesthetic issue is warranted in their CCR.

Understanding the thought processes, or mental models, for both experts and consumers is extremely important in crafting an effective CCR. Since the knowledge gap is so wide between experts and consumers, CWSs need to identify what consumers know at the outset, correct misinformation, and provide accurate information as a result (Löfstedt & Frewer 1998).

CCRs are health-based communication documents because the USEPA views CCRs as a means to improve public health through educational materials (USEPA 2009). If the CCR is serving as a public health communication, it is important to recognize potential parallels between the CWS/consumer and the doctor/patient relationships, including: ‘(1) presenting technical and scientific information; (2) presenting possible traumatic information; (3) presenting risks, probabilities, and uncertainties as experts are relied upon to provide clear, unbiased, and value-free information and professional recommendations’ (Boyle & Holtgrave 1989). Just as doctors inform their patients, CCRs present information about drinking water that allows consumers to make educated decisions to drink the water that may or may not affect their health (USEPA 2009, 2012a). In fact, USEPA mandated language for the immune-compromised population encourages these individuals to seek advice about drinking water from their health care providers (USEPA 2009).

Additionally, communications should be more than one-time messages or pamphlets; there needs to be continuing dialogue with open channels of discussions, including talks about uncertainty (United States President's Commission for the Study of Ethical Problems in Medicine, Biomedical and Behavioral Research 1982; Boyle & Holtgrave 1989). For CCRs, the dialogue should be between the CWS and the consumer, and require that CWSs encourage consumer feedback (Dietrich et al. 2014), in part through widely publicized contact information for telephone, email, and social media. The American Water Works Association (AWWA) Research Foundation established that consumers expect more water quality information and CWSs must meet consumer's expectations through routine dialogue (Means et al. 2002).

CCRs provide an opportunity for effective communication to address consumer concerns and build trust by promoting consumer confidence (Meyer-Emerick 2004). Thus, when an unexpected contamination event occurs that affects drinking water quality and consumer confidence, the annual CCR can provide a mechanism for communicating with consumers to acknowledge the event and to maintain or rebuild trust and confidence. This was done by the City of Danville, VA when an estimated 39,000 ton of coal spilled into its water supply, the Dan River, in February 2014 (Gabriel 2014). The 2014 CCR (City of Danville 2015) described the effects of the spill on the CWS water quality, treatment, and monitoring. The 2014 CCR also acknowledged a taste-and-odor event that occurred in February and March of 2015 (Morrison 2015). The taste-and-odor event was linked to an algal bloom and the CCR described the CWS's effort to identify the cause (City of Danville 2015).

While some CWSs use their CCRs to communicate effectively about contamination events, other CWSs miss this opportunity. The severe cyanobacteria bloom in Lake Erie in August 2014 was met with proactive microcystin monitoring by the City of Toledo, Ohio. When the cyanotoxic microcystin was detected, public notification of ‘do not use’ the water for drinking or cooking was issued and monitoring continued (Arenschield 2014). Yet, the 2014 CCR (City of Toledo 2015) only referred to the ‘water event’ without explanation and provided monitoring data for microcystin without referring to the severe algal bloom. Furthermore, lingering concerns of West Virginia consumers affected by the crude 4-methlycyclohexane methanol (MCHM) industrial chemical spill in January 2014 existed into 2015 (Dizard 2015). Nonetheless, the July 2014 CCR (for year 2013) and July 2015 CCR (for year 2014) only slightly mentioned the chemical spill that resulted in issuance of ‘do not use’ water orders for about 300,000 residents near Charleston, WV (Howard 2014; Manuel 2014). For several days during and after the spill, a pervasive sweet licorice odor occurred; it continued for months with occasional odor episodes (Foreman et al. 2015; Gallagher et al. 2015; Sain et al. 2015).

Unfortunately, there was a lack of effective risk communication during the crisis (Parkin et al. 2006), causing more anxiety than confidence in the Charleston area CWS (Manuel 2014; Schade et al. 2015). The 2013 CCR issued in July 2014 could have been an ideal opportunity to reach out to the public and rebuild confidence in the drinking water, but was little used to explain the cause, effect, and lingering issues associated with the odorous MCHM incident (West Virginia American Water 2014). Likewise, the 2014 CCR issued in July 2015 only presented summary water quality monitoring data for MCHM and did not provide educational information about effects of the spill or address consumer concerns (West Virginia American Water 2015a) as would be expected if a CCR is to: ‘Improve public health protection by providing educational material to allow consumers to make educated decisions regarding any potential health risks pertaining to the quality, treatment, and management of their drinking water supply’ (USEPA 2009). Instead, educational information and customer service communications and a video from the CWS on the MCHM spill can be found on the West Virginia American Water (2015b) website. The CCRs issued in July 2014 or 2015 would have been ideal opportunities to present this message or reference the website.

Changes to CCRs in all seven areas of the Index will greatly improve the quality of communication for the main message and the sub-messages, and CWSs could benefit from improved consumer understanding and possible satisfaction in their drinking water. The USEPA and CWSs should utilize the CDC's Index to improve the standardized scripts such as those for as lead, copper, immuno-compromised individuals, and water conservation in the CCR-iWriter software. CWSs should continue to utilize the CCR-iWriter tool, but should also improve their individualized CCR by implementing the CDC's Index tool and utility personnel's own writings to better communicate information related to their water system and constituent consumer base, i.e. source water, chemical spills, contaminants, exceedances, etc.

Consumers desire information about drinking water quality, and CWSs, with USEPA guidance, generate ‘water quality reports’ intended to meet their consumers’ needs. This assessment of mandated CCR documents demonstrated that CCRs are not adequate in informing consumers about the safety of their drinking water. The 30 CCRs sampled nationwide did not receive passing scores on the CDC's Clear Communication Index. Only 63% directly informed their consumers whether or not the water was safe to drink according to state and federal standards. The goal of the CCR is to communicate information about drinking water, thus, CCRs should be written to clearly communicate to their intended consumer audience. Since the CCR rule was promulgated in 1999, new strategies and understandings of how to communicate science and risk to consumers have been developed and their implementation should be considered for improving the effectiveness of future CCRs.

Funding for this project was provided by the Virginia Tech Water Interface, Interdisciplinary Graduate Education Program and the Institute for Critical Technology and Applied Science. The authors appreciate the insights of Mr Alan Roberson of the American Water Works Association and CWS personnel who are members of the American Water Works Association.