After the January 2010 earthquake in Haiti, an existing development program promoting household water treatment with chlorine rapidly expanded and provided relief to 15,000 earthquake-affected households. Initially, 157 community health workers (CHWs) distributed chlorine tablets; ten months later, CHWs began selling locally manufactured solution. The program was externally evaluated in March and November 2010; 77–90% of recipients had free chlorine residual (FCR) in household water. Internal monitoring by three supervisors and 157 CHWs also began in 2010. We analyzed results from 9,832 supervisor and 80,371 CHW monitoring visits conducted between 2010 and 2014 to assess: whether success continued in the rehabilitation phase; internal data validity; and factors impacting adoption. In 2010, 72.7% of supervisor visits documented total chlorine residual (TCR) comparable to external evaluation results. TCR presence was associated with certain supervisors/CHWs, earlier program year and month (in 2014, supervisor visits TCR presence dropped to 52.1%), living in plains (not mountainous) regions, and certain calendar months. CHW visits recorded 18.1% higher TCR presence than supervisor visits, indicating bias. Our results document a program with sustained (although slightly declining) household chlorination use, provide insight into validity in internal monitoring, and inform discussions on the value of linking successful development programs to emergency relief, rehabilitation, and development.
Worldwide, approximately 663 million people lack access to an improved water source and an estimated 1.2 billion more rely on contaminated water sources (Onda et al. 2012; WHO/UNICEF 2015). Household water treatment and safe storage (HWTS) can be a cost-effective means of improving drinking water quality (Clasen et al. 2007) and reducing diarrheal disease in households where access to microbiologically safe water is limited (Clasen et al. 2006; Clasen 2015). HWTS is, therefore, recommended as part of a comprehensive strategy to prevent diarrheal disease in low-income settings without access to safe drinking water (WHO/UNICEF 2011).
In the 2012 Demographic Health Survey in Haiti, 88% of urban and 49% of rural respondents reported having access to an improved water source (Cayemittes et al. 2013). Additionally, 60% of urban and 78% of rural respondents self-reported use of HWTS options. Of those reporting HWT use, 96% reported using chlorine-based products. The percent of the population that had confirmed use of HWTS products was not assessed. In another 2012 survey of 1,024 households, 68.8% of rural respondents self-reported using HWTS, and 27.7% of household had stored water with free chlorine residual (FCR), a confirmation of water treatment (PSI 2012).
One chlorine-based HWTS program, the Safe Water System (SWS), was developed by the U.S. Centers for Disease Control and Prevention (CDC) and the Pan American Health Organization and consists of three components: (1) water treatment with sodium hypochlorite solution (chlorine); (2) safe storage of household drinking water; and (3) education and behavior change messages to encourage safe household water, sanitation, and hygiene practices (CDC 2008). Randomized, controlled trials have shown SWS use to reduce diarrhea by 22–84% over up to one year of use in children and adults (Clasen et al. 2006).
In 2001, an SWS project was started in Jolivert, a rural community in Northwest Haiti (DSI 2015). In 2008, the non-governmental organization Deep Springs International (DSI) began managing the Jolivert program, training Haitian technicians to: (1) manufacture quality-controlled chlorine solution, branded ‘Gadyen Dlo’ (‘Water Guardian’ in Haitian Kreyòl); (2) enroll participating families through the sale of safe storage containers consisting of modified buckets with lids and taps; (3) sell Gadyen Dlo to participating families; (4) maintain sales records for each participating family; and (5) conduct regular household visits to monitor Gadyen Dlo use and provide ongoing education.
In a 2008 external evaluation of the Jolivert program, 56% of participants (versus 10% of controls who were not program participants and only had access to other products available on the local market) had FCR in their drinking water and children of <5 years old had 59% reduced odds of diarrhea (95% CI = 0.21, 0.79) (Harshfield et al. 2012).
In 2008, DSI began replicating the Jolivert program in other Haitian communities, including a pilot in Léogâne. DSI rapidly scaled up this pilot as an emergency response activity after the January 2010 earthquake, as Léogâne was near the earthquake's epicenter. In total, about 15,000 families were provided a safe storage container by DSI, leading to post-emergency programmatic coverage of approximately 40% of families throughout Léogâne Commune (20% in the plains and 60–100% in the mountains). The program was, and is, coordinated by three Haitian DSI supervisors who support 157 local community health workers (CHW). For ten months immediately following the earthquake, Aquatabs™ chlorine tablets were distributed at no cost, after which DSI resumed normal sales of Gadyen Dlo. In 2010, DSI supervisors and CHWs began to conduct regular unannounced household monitoring visits to recipients to complete education, conduct a small survey, and measure chlorine residual in stored household drinking water.
In two external evaluations conducted in 2010, two and ten months post-earthquake, 77% and 90% of participants in this program had FCR in their drinking water, respectively (Lantagne & Clasen 2012, 2013). Please note the 10-month follow-up occurred one month after the introduction of cholera to Haiti, and it is not possible from the dataset to distinguish whether Aquatabs or Gadyen Dlo were used to result in the FCR. The DSI program was the most successful of 14 programs evaluated across four different emergency responses. In contrast, only 10–17% of households which received free chlorine tablets without follow-up had FCR in their drinking water in two studies conducted in 2010 and 2012 (Lantagne & Clasen 2012, Patrick et al. 2013). Program success was attributed to DSI providing an effective HWTS method, with the necessary supplies and training provided, to households with contaminated water which were familiar with the method before the emergency (Lantagne & Clasen 2012). These results suggest that, when sufficient training and follow-up is provided, chlorine can be used to improve drinking water quality and reduce the risk of diarrheal disease in the acute emergency and initial rehabilitation phases. However, external evaluations were not conducted as the emergency relief project progressed further into rehabilitation and development. Currently, there is significant interest in linking relief to rehabilitation and development (LRRD) in emergency response. However, there are few examples showing that emergency response water programs can successfully transition to rehabilitation/development programs (German WASH Network 2014).
At the time of writing, the DSI Léogâne program, and the CHW and supervisor monitoring, are ongoing. Extensive monitoring and evaluation data such as these are unique within HWTS and emergency programs. Such internal monitoring data are relatively less expensive and more rapid to obtain than external evaluation data, allowing for collection of a large volume of data; however, these data remain subject to conflict of interest and bias (UNDP 2009; IFRC 2011; Lantagne et al. 2012). External evaluation data provide a more objective dataset, although are more costly to obtain. Using data from multiple sources should be triangulated to minimize biases inherent in monitoring and evaluation, and objective verification indicators such as FCR and TCR provide a check for courtesy bias. Please note that both TCR and FCR are objective measures of whether chlorine has been used to treat water. TCR includes both FCR and the less effective disinfectant combined chlorine. Thus, TCR will always be equal to, or higher than, FCR values.
Thus, there are three sets of monitoring and evaluation data available from the DSI Léogâne program: (1) external evaluation data collected at two and ten months after program initiation; (2) DSI supervisor data collected from 2010 to 2014; and (3) local CHW data collected from 2010 to 2014. Please note that the external evaluation used FCR as an objective measure, and DSI used TCR. In our analysis, we aimed to: (1) assess whether documented programmatic success in the emergency response phase continued into the development phase (LRRD); (2) assess data internal validity by comparing the TCR data collected by supervisors and CHWs; and (3) analyze programmatic factors that lead to higher TCR rates.
Data were collected from households in Léogâne Commune, Haiti, during normal operations of the DSI Léogâne program. Before the earthquake, approximately 1,500 families had purchased Gadyen Dlo in the Léogâne DSI program. After the earthquake, three DSI supervisors trained 157 CHWs to first freely distribute Aquatabs during the post-earthquake emergency to approximately 15,000 families in Léogâne in 2010, and then to sell Gadyen Dlo to these same 15,000 families from 2010 onwards.
The CHWs distributed branded safe storage containers and chlorine products to a selection of households near their home. CHWs were instructed to provide the materials to households with the greatest need and/or young children; however, ultimately the CHWs decided which households would be program recipients. On average, the CHWs distributed approximately a total of 100 safe storage containers each, leading to post-emergency programmatic coverage of approximately 40% of families throughout Léogâne Commune (20% in the plains and 60–100% in the mountains).
The CHWs regularly visited households to distribute/sell the chlorine products, conduct training, conduct a small survey, and perform a TCR test of stored household water (if water was available). Each household should have been/be visited at least once every three months, although the exact visit schedule was determined by individual CHWs. DSI supervisors regularly visited each CHW and selected a sub-sample of households in their community for small survey and TCR testing. Initially, these visits were performed with the CHW, and afterwards performed independently. Supervisors were instructed to visit different communities served by the same CHW to ensure the entire region of each CHW was represented. Please note all household visits by both supervisors and CHWs were unannounced; although householders would have been aware that CHWs and supervisors would likely visit their home at some point in the future.
The outcome of interest was TCR, which was tested by both supervisors and CHWs using a colorimetric pool test kit using orthotolidine solution obtained from Blue Pool Supplies (Redwood City, CA, USA). Results were recorded as present if there was yellow color observed, or absent if no yellow color was observed; independent testing has confirmed that a yellow color is observed at 0.2 mg/L (Murray & Lantagne 2014). TCR presence indicates that the water was treated with chlorine and has sufficient chlorine present to ensure safety in storage by preventing recontamination.
Supervisor visit data analysis
DSI provided a Microsoft Excel spreadsheet including results from all supervisor visits conducted from 2010 to 2014. These data were cleaned and analyzed using Microsoft Excel 2011 (Redmond, WA, USA) and Stata 13.1 (College Station, TX, USA).
The data were initially analyzed across six variables: (1) supervisor; (2) program year; (3) program month; (4) calendar month; (5) season; and (6) geographical location. The dataset was then subsequently stratified by CHW, and analyzed for the same six variables. The seasonality variable was generated using publicly available weather data to determine rainy versus dry months (WBG 2015). The geographical location variable was coded as whether households were in a plains or mountainous region (using maps provided by DSI). The main outcome of interest was percent positive TCR. Of note, households in which water was not present at the time of the unannounced visits were considered to have a negative TCR result: (1) in order to maintain consistency between data sources; and (2) as the household did not have safe drinking water at the time of the unannounced visit, and therefore was not considered to be using the chlorine. Households in which no one was present at the time of the unannounced surveys were excluded from analysis.
Chi-square tests were used to assess differences in TCR test results (presence/absence) by supervisor, program year, calendar month, season, and geographical location and t-tests were used to assess differences by program month.
Additionally, to account for the fact we do not know if all data were independent (i.e., if households were visited multiple times), we recompleted all chi-square and t-tests using 16 three-month data segments (i.e., January–March 2010 (segment 1), April–June 2010 (segment 2)). We selected three-month sections because CHWs were asked to visit each household every three months as a minimum, and thus we could assume the number of repeated households in each three-month segment would be negligibly small. All results were analyzed using 0.05 as a cutoff for statistical significance.
CHW visit data
Each CHW recorded monthly visits on a paper form, submitted to supervisors at DSI. The paper forms for the time period 2010–2014 were mailed to Tufts University. Data were entered using Google Forms (Mountain View, CA, USA), exported into Microsoft Excel, and cleaned and analyzed using Stata 13.1. As with supervisor data, the main outcome of interest was the TCR test result; households that did not have water were considered to have a negative TCR result and households in which no one was present at the time of the unannounced surveys were excluded from analysis.
The percent positive TCR test result for all visits for each CHW was calculated. This variable in the CHW dataset was compared with the equivalent variable in the supervisor dataset. A scatter plot was generated to visualize the correlation, and a Wilcoxon signed-rank test was performed to quantify the correlation between supervisor and CHW datasets for this variable.
To validate the supervisor and CHW datasets, two additional analyses were completed: (1) the total volume of liquid chlorine sold, by quarter, from October 2010 to December 2014 was obtained and compared to the measured TCR; and (2) a scatter plot visualizing the correlation between average TCR in the CHW and supervisor datasets over time (with each dot one month of data).
All data were provided from DSI to Tufts University in de-identified format, and the Tufts University Institutional Review Board approved this secondary analysis.
From 2010 to 2014, three supervisors performed 9,832 household visits and 157 CHWs performed 80,371 household visits. Although the exact number of distinct households reached with visits is unknown as no household ID numbers were recorded, the maximum number of households visited was 15,000, the number of households in the project. Please remember that in external evaluations conducted in 2010 two and ten months post-earthquake, 77% and 90% of participants in this program had FCR in their drinking water, respectively (Lantagne & Clasen 2012, 2013). Please note the ten-month evaluation happened to occur one month after the introduction of cholera to Haiti.
Supervisor visit data – full dataset
Across the five program years from 2010 to 2014, the average percent of TCR positive test results ranged from 52.1 to 72.7% (Figure 1(b)). This result varied significantly by year (p < 0.001), with the proportion of positive tests highest during 2010 (72.7%) and lowest in 2014 (52.1%). The sample size was 3,080, 2,514, 848, 2,837, and 514 in the years 2010, 2011, 2012, 2013, and 2014, respectively.
Across the 47 program months from 2010 to 2014, the percent TCR positive ranged from 33.3 to 83.3% (Figure 1(d)). Percent positive TCR trended significantly downwards over time (p < 0.001). The sample size in the 47 months averaged 273, with a median of 237, a minimum of 8, and a maximum of 818.
When data across the 47 months were aggregated by 12 calendar months, the average positive TCR percentage varied significantly by calendar month, although with less range than program year and month, from 63.0 to 66.7% (p < 0.001) (Figure 1(c)). The sample size in the 12 calendar months averaged 816, with a median of 917, a minimum of 30, and a maximum of 1,698.
Overall, 5,851 samples were collected in the dry season and 3,869 samples collected in the rainy season, of which, 3,805 (66.0%) and 2,552 (65.0%) were TCR positive, respectively (Figure 1(b)). This difference was not statistically significant (p = 0.346). Overall, 2,383 samples came from households in a plains region, and 3,474 samples were from households in mountainous areas (please note there was a large number of samples without region noted). Households in the plains regions had a statistically significantly higher percentage of positive TCR test results (70.5%) compared to those households that came from mountainous areas (65.8%) (p < 0.001). Additionally, when data were stratified by geographical region, there was a negative trend in TCR presence by program month in mountainous regions (Pearson's correlation coefficient = −0.024, p = 0.16) and a positive trend in the homes of the plains regions (Pearson's correlation coefficient = 0.028, p = 0.18).
When the data were reanalyzed using three-month segments, there were three individual segments that were different from the results presented above: (1) in months 7–9 (segment 3), visit month was not significant (p = 0.42); (2) in months 10–12 (segment 4), visit month was not significant (p = 0.26); and (3) in months 10–12 (segment 4), geographical region was not significant (p = 0.18). In all others analyses, across all 16 segments, the results by segment were the same as the result across the whole dataset.
Supervisor data stratified by CHW
Monitoring and evaluation results from the DSI Léogâne SWS program – which existed before the earthquake, rapidly expanded following the earthquake, and continues as of writing – document that: (1) 77% of recipient households surveyed in an external evaluation two months after the earthquake had FCR; (2) 90% of households surveyed in external evaluation conducted ten months after the earthquake (and at the start of the cholera outbreak) had FCR; (3) 72.7% of all DSI supervisor monitoring visit tests in 2010 had TCR; and (4) 52.1% of DSI supervisor monitoring visit tests in 2014 had TCR. Overall, the external evaluation and supervisor internal monitoring data describe a successful relief to rehabilitation and development water treatment program, with ongoing (although slightly declining) use of chlorine over time. These data are validated by DSI sales records, showing sales continuing into 2013 and 2014. However, these data also raise interesting questions around the topics of: (1) validity of various sources of monitoring and evaluation data; (2) the importance of individual CHWs and supervisors to program success metrics; (3) sustained use of chlorination, and the decline in TCR over time; and (4) lessons learned in developing and conducting a relief-to-rehabilitation water program.
We found rough consistency between the external evaluation and supervisor monitoring data collected in 2010 (77–90% compared to 72.7%), with both sets of data documenting programmatic success in the emergency response phase, and continued success ongoing. It is hypothesized that the very high 90% FCR rate found in the external evaluation in November 2010 might have been due to increased perception of risk in the immediate aftermath of the introduction of cholera to Haiti, which occurred in October 2010. Additionally, when we graphed supervisor percent positive TCR and CHW percent positive TCR by program month (Figure 4(b)), we saw consistency between the results over time.
However, the CHW monitoring data had, on average, 18.1% higher TCR presence than the supervisor data, indicating low internal validity and bias, and highlights the need to complete data triangulation (UNDP 2009; IFRC 2011). This low internal validity is not unexpected, as CHWs implementing the program might have conscious or unconscious selection bias in selecting households they know have TCR to increase their success metrics and/or that they visit more often or are easier to visit, and thus are more likely to have TCR. While these biases exist, this does not indicate that CHW's visit should be discontinued, as continued education on the products conducted during household visits likely is a key factor in sustained adoption (Figueroa & Kincaid 2010). This inflation of program success metrics is the reason it is recommended that individuals are not involved in programmatic implementation conduct evaluation (UNDP 2009; IFRC 2011). A limitation of monitoring is that it is often conducted by those directly involved in program implementation, and thus subject to these biases (UNDP 2009; IFRC 2011).
TCR presence was correlated with certain supervisors and CHWs. Differences found across the three supervisors may be because of supervisor performance in encouraging CHWs to conduct high-quality outreach, or simply that each supervisor consistently oversaw a third of CHWs and there were differences in the CHW cohorts or geographical areas the supervisors oversaw. There was high variability between CHW data in the supervisor dataset, with a mean of 65% TCR by CHW and a range of 29–96%. Individual CHWs could have had different dedication or ability to conduct their responsibilities, including conducting visits, completing education with recipients, collecting data, and distributing or selling the chlorine products. Initially, CHWs were paid only for completing reports; they then transitioned to receiving a proportion of their compensation as a percentage of bottle sales. The CHW is the person who interfaced with recipients, and thus CHW dedication and ability is key to program success.
TCR declined over time in the program, which may reflect: (1) increased perception of risk during the emergency (particularly immediately after the cholera emergency in the ten-month evaluation in November 2010 that found 90% FCR), and the decline of perceived risk when not in an acute emergency; and (2) the shift from free distribution of Aquatabs during 2010 back to sales of Gadyen Dlo during 2011–2014. Another consideration is that recipients who had received a DSI bucket may have been using an alternate source of chlorine, such as chlorine tablets of varying appropriate and inappropriate doses distributed by another NGO or locally purchased chlorine powders available in the market (Harshfield et al. 2012). The decline in TCR presence over time was primarily in households in mountainous regions, which might reflect less access to chlorination products. However, >50% of households had TCR presence in the fifth year of the program, which is a much higher rate than previously documented for sustained use of chlorination (Hunter 2009).
Previously documented successful relief to rehabilitation and development programs had the characteristics of being first implemented in an emergency, and then transitioning to long-term holistic programs operating in stable post-emergency contexts with consistent staffing and community input and management (Hyder 1996; Macrae et al. 1997; Maxwell 1999; Aubee & Hussein 2002; House 2007). The Léogâne program had all these aspects, but had three important additional pieces: (1) it was operational before the emergency (in large-scale programming elsewhere in Haiti and pilot phase programming in Léogâne), and thus all resources and staffing were locally available before the emergency; (2) an adaptive strategy of product distribution was used, from sales-based Gadyen Dlo bottles to free distribution tablets (which were very well known in Haiti before the earthquake), back to sales-based Gadyen Dlo bottles; and (3) DSI maintained collection of consistent TCR data through and after the emergency phase. Rather than LRRD, this is an example of linking a development program to relief and rehabilitation and back to development. To accomplish this, DSI was funded by small donations to start the Léogâne program (in partnership with an organization focusing on malnutrition), then received a large grant from an emergency response organization to scale up after the earthquake, and has continued the program using income from Gadyen Dlo sales, grants, and donations.
In literature discussing the strengths and limitations of LRRD, it has been stated that a ‘more holistic approach to assistance linking short-term relief measures with longer-term development programmes in order to create synergies and provide a more sustainable response to crisis situations’ is necessary (German WASH Network 2014). This linking of short-term relief measures with longer-term development programs is exactly what DSI completed in Haiti. DSI's results are not unique, as an evaluation of the Netherlands’ funded Haiti earthquake response found that organizations working in development in Haiti before the earthquake could switch from a development mode to an emergency mode ‘without major difficulties’ (IOB 2013). Based on these results, it is recommended emergency organizations consider partnering with successful development organizations to ensure program sustainability (German WASH Network 2014). In reality, however, there are often financial and institutional barriers to these partnerships (Imanishi et al. 2014).
The limitations of this work include: (1) households were not selected randomly; ultimately, individual supervisors and CHWs selected households to visit, which is likely to over-represent use (although could also under-represent use); (2) data from the same household were collected multiple times, yielding clustered data and the possibility that individual households are over-represented or under-represented; (3) it is unknown how many distinct houses were visited, and while we accounted for this with the three-month segment analysis, there may remain some error; (4) some CHWs reported no CHW data, and this may under-represent (or over-represent) use; (5) the classification of households without water (∼4% of total households) as having no TCR may bias the results towards the null, and under-represent use; (6) logistic regression was not valid given the questions surrounding data independence, which limits our understanding of factors that led to uptake; (7) there were missing data (particularly in the region analysis); and (8) cross-sectional external evaluations were compared with time-averaged internal evaluations. These limitations come from using a real-world monitoring dataset (as opposed to a controlled trial dataset), and do not diminish the value of the results. These data need to be interpreted with an understanding of these real-world limitations. However, it should be noted that there was consistency between the external and supervisor data in the initial sampling months, consistency between the supervisor and CHW data (by supervisor) as seen when comparing Figures 1(a) and 2(b), consistency between monitoring results and program sales (as seen in Figure 4(a)), and consistency between supervisor and CHW data across the program months (as seen in Figure 4(b)). These consistencies indicate that there is some validity in the dataset. Lastly, given the data presented herein, it is recommended to complete a mixed-method case study evaluation of this program to document lessons learned in establishing a successful relief-to-rehabilitation program.
Although there was decline in TCR over five years, the overall rate of positive TCR test results is higher than documented in other SWS projects. These data provide insight into how to develop a sustainable, long-term HWTS chlorination program that can expand to respond to emergencies and progress through relief back into development. As such, continuing to conduct systematic supervisor internal monitoring – and responding to that monitoring to target areas with declining use rates with additional resources – is recommended. CHW monitoring is recommended because conducting training and household visits encourage adoption.
Our results provide insight into validity in internal monitoring, document a program with sustained use of HWTS chlorination (although slightly declining over time), and inform discussions on linking successful development programs to emergency relief, rehabilitation, and development.
This work was completed with partial support from the Centers for Disease Control and Prevention (for D. Lantagne, salary via a consulting contract) and by the National Science Foundation (0966093) (for M. Ritter, salary). The findings and conclusions in this publication are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.