Abstract
As a 2010 earthquake and cholera response project, in 2013–2014, an international non-governmental organization, working with local organizations, installed water systems in two Haitian communities. In 2016, 2 years after installation, we conducted a mixed-methods evaluation to assess sustained effectiveness, including 20 infrastructure assessments, 199 household surveys, water quality testing, 30 key informant interviews, and financial analysis of two water committee records. One of 14 (7.1%) installed kiosks were functional during the evaluation, with 42–60% of survey respondents near installed sources reporting using system water for drinking. No household water sample had detectable chlorine residual and non-detectable E. coli, except when household water treatment was reported. Informants expressed appreciation for the project, and discussed difficulties with operations, maintenance, and community engagement. Revenue was initially sufficient for operations and maintenance, although not sustainable because lack of trust led to poor quality service and then lack of payment. While the functional kiosks did provide water to some residents, overall project sustained effectiveness was not reached due to operational and maintenance issues, and lack of governance and community engagement. We recommend future implementers of small-scale water systems in complex contexts like Haiti plan for the technical and social components necessary for sustained effectiveness.
HIGHLIGHTS
Evaluating sustained effectiveness of water infrastructure interventions in rural Haiti.
Identifying factors that could improve small-scale water infrastructure/water system management in complex contexts.
Financial analyses of water infrastructure implementation.
INTRODUCTION
Investments in drinking water infrastructure are crucial to prevent transmission of diarrheal diseases, which, in 2019, were the third leading cause of death in children <5 (IHME 2019). Despite the known benefits of safely managed water, worldwide, in 2020, one in four people lacked safely managed drinking water (defined as water from an improved source (e.g. household connections, public standpipes, boreholes), located on premises, available when needed, and free from fecal and chemical contamination) (WHO/UNICEF 2020). The remaining 25% of the world's population uses lower tiers of service, including ‘basic’ (water from an improved source less than 30 min away); ‘limited’ (water from an improved source more than 30 min away); and, ‘unimproved’ (water from an unprotected well or spring).
In Haiti, specifically, data on safely managed water sources in 2020 are not available, although 77% of the population were estimated to use improved sources in 2020 (WHO/UNICEF 2020). This lack of access to safely managed water is partially attributed to the fact that, in 2010, two major emergencies occurred in Haiti, an earthquake on January 12, 2010, that killed over 200,000 and left another 2 million people homeless, and a cholera outbreak that began in October 2010 and has since infected over 820,000 and killed nearly 10,000 people (Gelting et al. 2013; Le Ministère de la Santé Publique et de la Population (MSPP) 2020).
Following these two 2010 emergencies, there was significant international interest and response in providing safe water and sanitation to the Haitian population. The humanitarian response to the combined Haitian emergencies led to more than 50 million US dollars investments in water, sanitation, and hygiene (WASH) interventions between 2011 and 2012, and over 100 non-governmental organizations (NGOs) implementing WASH programs were identified in Haiti in 2011 (Financial Tracking Services 2011; Gelting et al. 2013).
Therefore, between November 2013 and August 2014, with funding from Cartier Philanthropy (previously Cartier Charitable Foundation), an international non-governmental organization (INGO) completed a water, sanitation, and hygiene (WASH) program that included water source protection, water supply infrastructure development, centralized water treatment, and additional WASH activities in two rural communities in Haiti that lacked safely managed water at the time. Cartier Philanthropy framed this programming as an emergency response project related to the 2010 Haitian earthquake with the goal of improving access to clean water and sanitation. Of note, in 2013–2014, when this water infrastructure program was implemented, 63.2% of the Haitian population had access to at least basic water (WHO/UNICEF 2020), and 5,055 deaths were attributed to diarrhea (IHME 2019). Additionally, although the SDGs and safely managed water were not the current metrics, it was known that these would be the metrics, and Cartier Philanthropy was interested in meeting long-term sustainable development goals.
In Community A, the INGO contracted with local organizations to build a gravity-fed water distribution system including a spring catchment, chlorination, and distribution via four kiosks. This system was intended to provide >7,500 people with safe drinking water. In Community B, a water distribution system was built with a filtration and chlorination system to provide more than 10,000 people with safe drinking water from 10 kiosks and 800 private connections. In both communities, water committees were established and trained with the goal of ensuring effective management and maintenance of water infrastructure. Additionally, it was planned that the recently established (in 2009) Haitian government agency La Direction Nationale de l'Eau Potable et de l'Assainissement (DINEPA) would oversee water system operations and management after project completion. Note the communities are not named herein to protect informant confidentiality, and are generally representative of communities needing safely managed water.
Achieving sustained effectiveness of water systems in low- and middle-income countries (LMICs) is difficult. Less sustainable systems are older, further from urban areas, and lack funding, accountability, regular maintenance, access to supplies, technical expertise, community management systems, functional water committees, gender equity in water committees, fee collection systems, trust between users and system operators, and user demand and usage (Foster 2013; Rayner et al. 2016a, 2016b). More sustainable water systems were ones including appropriate technologies, management schemes, fee collection, and regular repair and maintenance (Fisher et al. 2015; Bartram & Cronk 2017). In complex emergency situations specifically (such as in Haiti with successive disasters, epidemics, and political emergencies), successes in sustainable water infrastructure include training of mechanics, supporting water committees, selecting simple technologies, and providing support after initial implementation (ACF 2007). Lack of engagement with local institutions and local capacity building were identified as barriers to sustained effectiveness because they led to technical failures (in system management) and social failures (in not understanding social relationships and community dynamics).
In this evaluation, we assessed the sustained effectiveness of the Cartier Philanthropy project in 2016, 2 years after project completion in 2014, using both international and donor metrics.
METHODS
To assess sustained effectiveness, we conducted one mixed-method evaluation, including infrastructure assessments, household surveys, key informant interviews (KIIs), and financial analysis. Data were triangulated across mixed methods in analysis to determine international and donor metrics were met. This evaluation was approved by the Tufts University (1516-20) and Haitian Institutional Review Boards (1512007). Written informed consent was obtained before each key informant interview and household survey.
Infrastructure assessments
In Communities A and B, each source, reservoir, and kiosk were visited with a community member informant involved with installation and assessed. The following data were collected: global positioning system (GPS) point, system operational capacity and condition data, availability of spare parts, consumables information, and water samples from kiosks when available.
Household surveys
Household surveys were carried out in Haitian Creole by trained Haitian enumerators. In each community, enumerators selected houses by beginning at a kiosk, then visiting every other house until 10 household surveys were completed (maximum 140 households). In Community B, an additional 80 households were randomly selected from a list of 228 households that paid private connection deposits (maximum 80 households). At each house, a survey was conducted, consisting of 77 questions on demographics, drinking water knowledge and practices, drinking water access and storage, and knowledge about the project. Enumerators followed a survey guide with close-ended questions that offered several options to choose from, questions that required enumerators to observe participants' actions after cues were given (i.e. ‘Can you show me where you wash your hands?’), and open-ended questions with associated probes. During the survey, a sample of stored household drinking water was collected. Survey data were entered into Microsoft Excel (Redmond, WA, USA) and analyzed using Stata 10.1 (College Station, TX, USA).
Water quality testing
Water samples from kiosks and households were analyzed for free chlorine residual (FCR) and E. coli. FCR was tested using a Hach® ColorWheel Test Kit (range 0.0–3.5 mg/L) and DPD-1 tablets (Loveland, CO, USA). Results ≥0.2 mg/L were considered to meet international guidelines (WHO 2011). To test E. coli, 100 mL samples were collected aseptically in Whirl-Pak™ bags with sodium thiosulfate (Nasco, Ft. Atkinson, WI, USA), stored on ice, and tested within 12 h of collection. Samples were tested using membrane filtration: sample water was diluted appropriately with sterile buffered water and then filtered aseptically through a 45-μm Millipore filter (Billerica, MA, USA) using a portable Millipore filtration stand. Filters were then placed in a plastic petri dish with a mColiBlue24® media soaked pad, and incubated at 35–37 °C for 24 h in a portable incubator. For quality control, sterile buffered water controls were tested every 20 plates, and 10% of samples were duplicated. Results were converted to CFU (colony forming units)/100 mL, and categorized according to World Health Organization (WHO) risk categories (WHO 2011).
Key informant interviews
KIIs were conducted with persons involved in project implementation, operation, and management, including members of the Water Committees. Interviews were semi-structured, with the interview guide consisting of 31 open-ended questions including personal background, water and sanitation knowledge, role in the project, opinions about project implementation, and capacity building activities of the project. Interview responses were recorded on paper forms, entered into Microsoft Excel, and coded into emergent themes in Excel by staff who speak Haitian Creole fluently. Key quotations were extracted and included herein.
Financial analysis
During KIIs, water committee financial reports were collected. Data were analyzed for revenue compared with expenses, revenue source distribution, and cash balance at the last report.
Sustained effectiveness
To assess sustained effectiveness, outcome metrics (co-developed between Cartier Philanthropy and Tufts University) were assessed, including relevance, effectiveness, efficiency, sustainability, and impact. Relevance was measured using responses from KIIs and the percentage of households using water provided by the system (from household surveys). Effectiveness was measured using infrastructure assessments to determine what was operational with detectable FCR and no detectable E. coli at the time of the survey. Efficiency was measured using household surveys as the percentage of households using water provided by the system with detectable FCR and no detectable E. coli. Sustainability was measured as the (1) percentage of infrastructure with sufficient spare parts, consumables, and ongoing operation and maintenance; (2) percentage of communities with operational water committees (KIIs, financial analysis), (3) percentage of communities with operational bank accounts with user fees (financial analysis), (4) percentage of communities with DINEPA or Mayoral involvement or support (KIIs), and (5) local organization empowerment as a result of program (KIIs). Impact was intended to be measured as improvements in water quality and health outcomes.
RESULTS
From March 22 to April 19, 2016, we carried out 20 infrastructure assessments, 199 household surveys, 30 KIIs, and collected two water committee financial report assessments. Results are presented below.
Infrastructure assessments
In Community A, a water distribution system was installed, including a capped spring flowing into a concrete reservoir (both in good condition) then through a distribution system to four kiosks. During the assessment, one of four kiosks (25%) were operational, and the one operational kiosk lacked stock of sufficient spare parts and chlorine consumables. Informants who visited the infrastructure systems with researchers stated that: two of the non-operational kiosks would function if valves were opened; there was at least one kiosk where water never reached due to potential unsound hydraulic system design; and/or, kiosks were closed because the population stopped paying for water and/or kiosk operators were not receiving their monthly stipend. Sample water tested from the operational kiosk had non-detectable FCR and was ‘medium risk’, with 21 E. coli CFU/100 mL.
In Community B, a spring box was built in the mountains, and water flowed via a sand filtration system and reservoir with a chlorination system into the community. The spring box and reservoir infrastructure appeared in good condition; however, the sand filtration system had not been recently cleaned. The distribution system was designed to provide water for 10 kiosks and household connections. No kiosks (0%) were operational during the evaluation because: water did not reach the kiosks; there were broken pipes and/or tanks; and/or, they were not opened by operators. Thus, no water samples were collected. Additionally, the non-operational kiosks also lacked sufficient spare parts and chlorine consumables in stock. Informants stated that 7 of 10 (70%) kiosks could be operational as is, or with minor repairs, that pressure decreased once private connections were installed, that there was superfluous air in the water system, and that spring box water was diverted to supply ‘The Fountain’, a water point for mountainous populations. The Fountain had continuous water flow, was in good condition except for a broken PVC pipe, and had non-detectable FCR and non-detectable E. coli.
Household surveys
Demographics
Overall, 199 households surveys were conducted, including 10 around each kiosk in Community A (40) and Community B (100), and 59 private connections in Community B-private. Average educational attainment was 5.7, 7.6, and 8.3 years in Communities A, B, and B-private, respectively (Table 1). Household cell phone ownership was 48% in Community A, and 92–97% in Communities B and B-private.
Demographics, WASH knowledge and practices, and water access
Community . | A (n = 40) . | B (n = 100) . | B-private (n = 59) . |
---|---|---|---|
Demographics | |||
Mean (SD) people per household | 5.1 (2.4) | 6.9 (3.5) | 5.2 (2.4) |
Female respondent, % (n) | 48% (40) | 72% (100) | 56% (59) |
Respondent attended school, % (n) | 58% (40) | 81% (99) | 95% (59) |
Mean (SD) highest grade, (n) | 5.7 (2.4) (23) | 7.6 (3.1) (75) | 8.3 (3.7) (47) |
Reports female head-of-household can read, % (n) | 35% (40) | 57% (100) | 81% (59) |
Reports male head-of-household can read, % (n) | 48% (40) | 60% (100) | 76% (59) |
Connected to electrical grid, % (n) | 0% (40) | 56% (99) | 91% (56) |
Has at least one cellular phone, % (n) | 48% (40) | 92% (92) | 97% (59) |
Has at least one motorcycle, % (n) | 0% (40) | 15% (92) | 44% (59) |
WASH Knowledge and Practices | |||
Believes water can make you sick, % (n) | 100% (37) | 94% (99) | 97% (59) |
Believes water in house is safe to drink, % (n) | 97% (37) | 92% (93) | 88% (58) |
Top reported ways to prevent diarrhea | |||
Drink safe water, % (n) | 50% (40) | 76% (100) | 80% (59) |
Wash hands, % (n) | 35% (40) | 54% (100) | 47% (59) |
I don't know, % (n) | 40% (40) | 14% (100) | 10% (59) |
Has on plot or shared latrine, % (n) | 28% (40) | 94% (100) | 98% (59) |
Has dedicated hand washing place, % (n) | 0% (39) | 9% (96) | 31% (51) |
Reports washes hands after using toilet, % (n) | 40% (40) | 73% (100) | 75% (59) |
Water Access, Treatment, and Storage | |||
Water storage container covered, % (n) | 85% (34) | 93% (98) | 92% (53) |
Provided drinking water sample, % (n) | 75% (40) | 93% (100) | 85% (59) |
Households with drinking water from system, % (n) | 60% (40) | 58% (100) | 42% (59) |
Reported treated at household, % (n) | 17% (30) | 23% (93) | 18% (50) |
With chlorine product, % (n) | 80% (5) | 95% (21) | 100% (9) |
With filter, % (n) | 0% (5) | 5% (21) | 0% (9) |
Don't know/Other, % (n) | 20% (5) | 0% (21) | 0% (9) |
Households with drinking/non-drinking water from system % (n) | 63% (40) | 72% (100) | 92% (59) |
System Use | |||
Report paying for water, % (n) | 20% (40) | 45% (100) | 90% (59) |
Mean monthly payment for water (SD), n | 25 (0) n = 8 | 157 (60.0) n = 36 | 199 (53.3) n = 53 |
Received information about the system, % (n) | 50% (40) | 35% (100) | 48% (59) |
Could name contact for questions, % (n) | 26% (38) | 33% (96) | 60% (53) |
Heard about hygiene activities, % (n) | 43% (40) | 57% (98) | 45% (58) |
Participated in hygiene activities, % (n) | 28% (40) | 7% (99) | 12% (58) |
System Perceptions | |||
Comments about project | |||
It's a good thing, % (n) | 45% (40) | 43% (100) | 41% (59) |
Before we didn't have access to water, % (n) | 22% (40) | 33% (100) | 25% (59) |
The source is protected now, % (n) | 38% (40) | 10% (100) | 0% (59) |
The water is treated, % (n) | 22% (40) | 10% (100) | 0% (59) |
Water isn't treated/safe, % (n) | 16% (40) | 14% (100) | 30% (59) |
It's not well managed, % (n) | 22% (40) | 29% (100) | 25% (59) |
A distribution schedule is needed, % (n) | 0% (40) | 22% (100) | 28% (59) |
Water is not given fairly, % (n) | 0% (40) | 20% (100) | 18% (59) |
Distribution is not frequent enough, % (n) | 0% (40) | 44% (100) | 46% (59) |
It's too expensive, % (n) | 22% (40) | 14% (100) | 25% (59) |
Community . | A (n = 40) . | B (n = 100) . | B-private (n = 59) . |
---|---|---|---|
Demographics | |||
Mean (SD) people per household | 5.1 (2.4) | 6.9 (3.5) | 5.2 (2.4) |
Female respondent, % (n) | 48% (40) | 72% (100) | 56% (59) |
Respondent attended school, % (n) | 58% (40) | 81% (99) | 95% (59) |
Mean (SD) highest grade, (n) | 5.7 (2.4) (23) | 7.6 (3.1) (75) | 8.3 (3.7) (47) |
Reports female head-of-household can read, % (n) | 35% (40) | 57% (100) | 81% (59) |
Reports male head-of-household can read, % (n) | 48% (40) | 60% (100) | 76% (59) |
Connected to electrical grid, % (n) | 0% (40) | 56% (99) | 91% (56) |
Has at least one cellular phone, % (n) | 48% (40) | 92% (92) | 97% (59) |
Has at least one motorcycle, % (n) | 0% (40) | 15% (92) | 44% (59) |
WASH Knowledge and Practices | |||
Believes water can make you sick, % (n) | 100% (37) | 94% (99) | 97% (59) |
Believes water in house is safe to drink, % (n) | 97% (37) | 92% (93) | 88% (58) |
Top reported ways to prevent diarrhea | |||
Drink safe water, % (n) | 50% (40) | 76% (100) | 80% (59) |
Wash hands, % (n) | 35% (40) | 54% (100) | 47% (59) |
I don't know, % (n) | 40% (40) | 14% (100) | 10% (59) |
Has on plot or shared latrine, % (n) | 28% (40) | 94% (100) | 98% (59) |
Has dedicated hand washing place, % (n) | 0% (39) | 9% (96) | 31% (51) |
Reports washes hands after using toilet, % (n) | 40% (40) | 73% (100) | 75% (59) |
Water Access, Treatment, and Storage | |||
Water storage container covered, % (n) | 85% (34) | 93% (98) | 92% (53) |
Provided drinking water sample, % (n) | 75% (40) | 93% (100) | 85% (59) |
Households with drinking water from system, % (n) | 60% (40) | 58% (100) | 42% (59) |
Reported treated at household, % (n) | 17% (30) | 23% (93) | 18% (50) |
With chlorine product, % (n) | 80% (5) | 95% (21) | 100% (9) |
With filter, % (n) | 0% (5) | 5% (21) | 0% (9) |
Don't know/Other, % (n) | 20% (5) | 0% (21) | 0% (9) |
Households with drinking/non-drinking water from system % (n) | 63% (40) | 72% (100) | 92% (59) |
System Use | |||
Report paying for water, % (n) | 20% (40) | 45% (100) | 90% (59) |
Mean monthly payment for water (SD), n | 25 (0) n = 8 | 157 (60.0) n = 36 | 199 (53.3) n = 53 |
Received information about the system, % (n) | 50% (40) | 35% (100) | 48% (59) |
Could name contact for questions, % (n) | 26% (38) | 33% (96) | 60% (53) |
Heard about hygiene activities, % (n) | 43% (40) | 57% (98) | 45% (58) |
Participated in hygiene activities, % (n) | 28% (40) | 7% (99) | 12% (58) |
System Perceptions | |||
Comments about project | |||
It's a good thing, % (n) | 45% (40) | 43% (100) | 41% (59) |
Before we didn't have access to water, % (n) | 22% (40) | 33% (100) | 25% (59) |
The source is protected now, % (n) | 38% (40) | 10% (100) | 0% (59) |
The water is treated, % (n) | 22% (40) | 10% (100) | 0% (59) |
Water isn't treated/safe, % (n) | 16% (40) | 14% (100) | 30% (59) |
It's not well managed, % (n) | 22% (40) | 29% (100) | 25% (59) |
A distribution schedule is needed, % (n) | 0% (40) | 22% (100) | 28% (59) |
Water is not given fairly, % (n) | 0% (40) | 20% (100) | 18% (59) |
Distribution is not frequent enough, % (n) | 0% (40) | 44% (100) | 46% (59) |
It's too expensive, % (n) | 22% (40) | 14% (100) | 25% (59) |
WASH knowledge and practices
Most respondents knew water can make you sick (94–100%), thought their water was safe to drink (88–97%), and covered their stored water (85–93%) (Table 1). Access to on-plot or shared latrines was 28% in Community A, and 94–98% in Communities B and B-private. Having a dedicated handwashing place was 0% in Community A, 9% in Community B, and 31% in Community B-private.
Water access, treatment, and storage
In total, 63, 72, and 92% of households had stored household INGO system water during the survey in Community A (from the single operational kiosk), B (from The Fountain or neighbors household connections), and B-private (from private connections), respectively (Table 1). However, fewer households, 60, 58, and 42%, reported using that water for drinking in Communities A, B, and B-private, respectively.
System use
In Community A, 20% of respondents reported paying an average 25 HTG (USD 0.40) per month for water (80% reported not paying) (Table 1). In Communities B and B-private, 45 and 90% of respondents reported paying an average 157 HTG (USD 2.60) and 199 HTG (USD 3.30), respectively. Overall, 14–25% of respondents reported water costs were too high.
In Communities A, B, and B-private, 50, 35, and 48% of respondents reported having received information about the INGO system, and 26, 33, and 60% could name someone to contact if they had questions, respectively (Table 1). Overall, 43, 57, and 45% of respondents had heard about hygiene activities, with 38, 7, and 12% reporting having participated, in Communities A, B, and B-private, respectively.
System perceptions
Overall, 45, 43, and 41% from Communities A, B, and B-private, respectively, reported believing that the INGO project was a good project, because there was no water before, the source is protected, and water is treated (Table 1). However, 14–30% of respondents stated the water is not safe and the system is not well managed, and in Communities B and B-private, respondents stated a distribution schedule is needed, distribution is not frequent enough, and water is not given fairly. 17, 23, and 18% of respondents reported treating their stored drinking water sample at home in Communities A, B, and B-private, respectively (Table 1). Of those respondents reporting treatment, 80, 95, and 100% of respondents, respectively, reported household chlorination.
Water quality
FCR was detected in 0% of household water samples in Community A, regardless of reported household water treatment. In Communities B and B-private, FCR was detected only in samples from households reporting household chlorination.
Geometric mean E. coli in household stored drinking water from the INGO system was 10.9, 33.5, and 9.5 CFU/100 mL without household treatment, and 11.3, 10.7, and 1.1 CFU/100 mL with reported household treatment, in Communities A, B, and B-private, respectively.
When categorized by WHO risk category, 4, 10, and 36% of household samples from the INGO system met the WHO guideline value of <1 E. coli CFU/100 mL in Communities A, B, and B-private water samples, respectively. An additional 42, 26, and 32% of household samples from the INGO system were <10 E. coli CFU/100 mL, leading to 46, 36, and 68% of household samples from the INGO system (in Communities A, B, and B-private, respectively) considered ‘low risk’.
Summary
Overall, survey results show that demographics were different across the study populations, with respondents in Communities B, and especially B-private, having higher education, knowledge, and indicators of wealth than in Community A. While 63, 72, and 92% of respondents in Communities A, B, and B-private could provide a sample of the INGO system water from their household during the evaluation, only 60, 58, and 42%, respectively, reported using that water for drinking, and a number of households treated that water before drinking. No sample from a household with INGO system water not treated with household water treatment met standards of safe drinking water: detectable FCR; and, non-detectable E. coli per 100 mL sample.
Key informant interviews
Nine themes emerged in KIIs with project and community leaders; each described below.
General feedback
Community A informants expressed appreciation for the project, stating the water system is too important to fall into disrepair, and they appreciated water being treated, improved access in remote areas, and employment opportunities. Community B informants said the water system was an ‘exceptional asset’ and a ‘gwo, gwo bagay’ (very, very big thing), and appreciated ‘anyone can drink the water’, employment opportunities, and time gained to earn money and attend school.
Engagement of authorities
In rural areas, DINEPA delegates water system management to water committees, and has guidelines on water committee establishment process, structure, and operation. In Community A, DINEPA was described as a ‘supplier of knowledge’, especially for leading water committee trainings. Informants also stated local political leaders were involved, but their lack of support was an implementation weakness. In Community B, DINEPA also led trainings and assisted water committees with activities (e.g. submitting reports). Informants stated the Mayor's office was involved, but lacked technical expertise, which prevented them from effectively engaging in the project.
Community engagement
In Community A, informants stated there was little community involvement in community meetings or to discuss water system maintenance. Informants expressed frustration that local workers with adequate skills were not hired for the project. In Community B, community members participated in constructing the water system, providing land, and attended community meetings.
Capacity building and empowerment
In Community A, the water committee received technical, management, public health, and social training from DINEPA and a local NGO. Informants expressed they did not receive sufficient training and material resources to function properly, stating the water committee was: ‘like children sent to school by their parents without being fed’ and ‘like orphans, like children without mothers and fathers’. In Community B, the water committee received accounting, management, and technical training from DINEPA and a local NGO. In Community B, informants reported acquiring useful skills in the trainings.
Water committees
Community A informants perceived the water committee as being ineffective at organizing community meetings and convincing the population to pay. An informant stated the current members ‘are there, just looking, not contributing actively’. Community B informants described the water committee as dysfunctional, with individual respondents stating ‘tèt anba’ (upside down), ‘kraze’ (broken), ‘jèsyon zero’ (zero management), and ‘the committee doesn't exist’. Both committees also reported significant inter-personal conflicts as well.
Operations and maintenance
Community A informants stated some residents were trained to operate the chlorinator and test FCR, and a kiosk operator was supplied chlorine tablets. During the evaluation, it had been a ‘while’ (but <1 year) since the system had been chlorinated, and 3–4 months since there had been chlorine tablets. In Community B, informants reported issues regarding who should cover the chlorine manager's transportation and chlorine purchase costs.
Recommendations
Community A informants recommended centering human relationships and community culture during project development, and having more time and input from local authorities. Community B informants recommended: hiring an engineer to address technical problems; clarifying who is responsible for specific costs; facilitating better communication between stakeholders; and, implementing different methods of encouraging people to pay for water.
Sustainable financing
In Community A, system operation was intended to be funded by user fees. Informants stated that, when the system was first implemented, about half the population paid. However, while it is Haitian policy that populations pay for water, some informants stated that: people did not want to, or could not afford to, pay. Informants attributed this to knowledge gaps and that water was free in nearby areas. In Community B, system operation was intended to be funded by user fees, but people with private connections gave free water to their neighbors. Additionally, people did not pay because they believed water should be free, the fee was too high, service quality was poor, and finances were perceived as mismanaged.
Financial analysis
Community B financial information, including: (a) revenue and expenses; (b) monthly income by source over time; and, (c) average revenue source and expense category.
Community B financial information, including: (a) revenue and expenses; (b) monthly income by source over time; and, (c) average revenue source and expense category.
Overall, for the time period, records were available, revenue was sufficient to operate and maintain the systems (when connection fees were included), as total revenue exceeded total expenses. Thus, there was some willingness to pay for the INGO systems, and lack of payments or decrease in payments occurred over time (as discussed in the KII section above) due to lack of trust in the water committee (including inter-personal conflicts), unwillingness (of some) to ever pay for water, and of the remaining population to continue to pay for unreliable water as system reliability declined.
Sustained effectiveness
Quantitative and qualitative data were analyzed for five key outcome metrics.
For relevance, in both communities, the installed infrastructure was highly appreciated. Positive comments by informants were the description as ‘gwo, gwo bagay’ (a very, very big thing) for the population and one informant stating, ‘I don't know how to say thank you to Cartier. I don't know where we would be today without them – we might not have water’. Overall, 63–92% of respondents had household stored water from the INGO system during the evaluation. However, only 42–60% of households used INGO system water for drinking, and many treated that water before drinking.
For effectiveness, only one of 14 kiosks was operational, and the Fountain and some household connections. It was unclear whether systems were designed correctly, and if all kiosks could be operational. No sample (0/2, 0%) from INGO system infrastructure met standards of having detectable FCR and non-detectible E. coli.
For efficiency, no sample (0/117, 0%) from a household with INGO system water that was not treated using household water treatment had detectable FCR and non-detectable E. coli.
For sustainability, neither INGO system (0/2, 0%) had sufficient spare parts, chlorine consumables in stock, an ongoing operation and maintenance plan, and/or an operational water committee. While in both communities, water management committees had been formed, received training, and created reports; neither water committee was fully functioning during the evaluation, and both reported within committee inter-personal conflicts. Additionally, both committees reported operational bank accounts, and analysis found sufficient fees collected to maintain average system maintenance for a time. However, lack of community trust in the water committee's ability to manage funds led to lack of sustainable fee collection and ongoing maintenance. Both INGO projects had local government involvement and support, and it was reported that local organizations were empowered by gaining project management skills such as prioritization and time management.
For impact, improvements in water quality and health outcomes were not able to be assessed.
DISCUSSION
Two years after systems were operational, in 2016, we carried out one mixed-methods evaluation to assess the sustained effectiveness of water systems implemented in Haiti in two communities, installed by a donor in response to the 2010 earthquake and cholera outbreak. We found that the projects were generally appreciated by key informant interviewees, and 42–60% of household survey respondents near the installed sources in Communities A, B, and B-private reporting relying on the INGO water system as their primary drinking source. However, neither system was reaching the populations as intended, and only one of four kiosks in Community A and none of ten kiosks in Community B were operational during the evaluation of two years after installation. Household connections and one alternate source were available in Community B. No sample met the WHO's safe drinking water guidelines of detectable FCR and non-detectable E. coli, except in households reporting chlorine-based household water treatment.
Operations and maintenance issues included both technical and systemic issues. Technical issues such as lack of water pressure and air in the system likely prevented water from reaching kiosks and private connections. Systemic issues included negative feedback loops stemming from lack of trust. Due to a feeling they should not need to pay for water, lack of trust in the water committee (including with financial management), and subsequent unreliability of the kiosk water distribution more and more community members did not feel they should pay for water, which led to supplies running short and undercompensated staff, which then led to poorer service and management, ultimately reinforcing community sentiments of not wanting to pay for unreliable services. The World Bank highlights this negative feedback loop in WASH interventions in LMIC, when water infrastructure implementers lack expertise and resources to reach sustained effectiveness, resulting in low-quality services at a price that consumers are not willing to pay (Goksu et al. 2019). This is exacerbated because DINEPA was founded in 2009, and while informants mentioned DINEPA provided value to the program, it is not clear that DINEPA could reasonably be expected to assume program operations. While the systems could have been brought to operation with minor repairs, and there was user payment for a time, the system did not meet initial high expectations and much of it fell into disrepair.
Sustained effectiveness of water infrastructure interventions is more difficult to achieve without adequate governance. Inadequate governance is common in complex and emergency-impacted communities in Haiti and South Sudan (Koski-Karell et al. 2016; Erismann et al. 2019). However, data from Bolivia, Ghana, Kenya, Peru, and Zambia shows, in the absence of governance, community engagement can be a means to implement successful small-scale water infrastructure interventions (Whittington et al. 2009; Kelly et al. 2017). When prioritized, community engagement can foster a sense of ownership, empowerment, and social capital, and create meaningful social and economic opportunities or community members, all of which can enable long-term sustainability (Kelly et al. 2017). Community engagement requires resources, time, and community trust, however, which can be difficult to achieve in complex emergency contexts like Haiti (Quintanilla 2015). Ongoing factors in Haiti such as extreme poverty, political volatility, government mistrust, and weak institutional capacity all likely impacted execution and sustainability of this project.
Our results are limited by lack of clinical records to assess whether water system installation lowered diarrheal disease rates. As shown in previous studies, it is difficult to conduct rigorous epidemiological evaluations of water supply improvements as people live, go to school, use water resources, and receive medical attention in different catchments (Matheson et al. 2012; deReil et al. 2018). Additionally, although to our knowledge this study was unique in evaluating actual water committee financial reports, individual bank account transactions were not available for analysis. Evaluating financial records of water infrastructure interventions can provide information about implementing sustainable revenue collection models (Jones 2010; Foster & Hope 2016). Lastly, funding was allocated only to evaluate this project once in 2016, so we could not compare our results to results from other projects implemented in Haiti during this time or this project across times. As this intervention was conducted after the 2010 earthquake and cholera outbreak in Haiti, the majority of assistance was allocated to short-term relief versus long-term projects. However, as this project was implemented and evaluated years after the earthquake, some may not consider this an earthquake-response project. Additionally, we feel these limitations did not impact the results of our study, and recommend further research on health impact of infrastructure and water committee financial management.
Based on our results and pre-existing literature, we recommended to Cartier Philanthropy, and we recommend to implementers and governments considering building infrastructure after emergencies in complex contexts, to: incorporate a budget for adaptations/contingencies into the initial project design to increase flexibility; conduct scoping evaluations to identify key stakeholders and develop sector and community engagement plans; invest heavily in community engagement and institutional capacity strengthening; budget for operations and maintenance; plan an exit strategy, including a transition phase where the system is operated and managed locally but still has external support; and, conduct monitoring and evaluation activities during and post-project.
These recommendations, which came from Haiti, are relevant to implementing infrastructure across complex emergency contexts, as the world strives to meet the SDGs by 2030, and provide safely managed drinking water for all.
ACKNOWLEDGEMENTS
We would like to thank the anonymous implementer of this project, who allowed us to evaluate the intervention, review the manuscript, and provide feedback.
DATA AVAILABILITY STATEMENT
All relevant data are included in the paper or its Supplementary Information.
CONFLICT OF INTEREST
The authors declare there is no conflict.