Monitoring and evaluation of WASH in schools programs: lessons from implementing organizations

Schools in low-income countries (LICs) are commonly built without access to safe drinking water, toilets, or handwashing stations (Ngales 2007). Similarly, school children in LICs often do not receive basic hygiene education in their national curriculum, and teachers are not equipped with the resources to provide this information to their students (Ngales 2007). Although research is limited on how many schools globally provide adequate water, sanitation and hygiene (WASH) conditions for their students, a survey of the United Nations Children's Fund (UNICEF) 60 priority WASH countries found only 50% of the countries provide adequate water and sanitation services to schoolchildren (UNICEF 2012). The importance of WASH in schools is increasing as evidenced by the proposed post-2015 Development Goals that are to be achieved by 2030. The specific WASH in school targets include: (1) all schools provide all users with handwashing and menstrual hygiene facilities; (2) all schools provide all users with adequate sanitation facilities; (3) all schools provide all users with basic drinking water supply. Additionally, there is a target, set for 2040, that the excreta from at least half of schools is safely managed.

Schools lacking adequate WASH conditions have been shown to have poorer health and learning outcomes among their pupils. One randomized control trial found that a WASH in schools intervention reduced the risk of diarrheal disease by 66% in the intervention group of students compared to controls (CARE 2012). Greater access to sustainable WASH in schools services is vital to improve health and educational outcomes in LICs (UNICEF 2011). An estimated 67 million school-aged children are not in school globally, of which 53% are girls (UNESCO 2011). When menstruation begins, girls are more likely to miss school due to insufficient WASH facilities and the privacy they provide. Studies show access to single-gender sanitation facilities can decrease these rates of absenteeism. A cross-sectional study in Nepal found 53% of interviewed female students had missed at least one day of school because of menstruation (WaterAid 2009). Positively, a cluster-randomized trial in Kenya found that hygiene promotion, water treatment, and sanitation access increased female attendance by 58% (Freeman et al. 2012). Jasper et al. reviewed the impacts of water and sanitation inadequacies in schools using 41 studies. The review documented demonstrates increased water consumption by pupils associated with improved access to water in schools, reduced diarrheal diseases among learners associated with better WASH facilities, and increased absenteeism when sanitation facilities were inadequate for menstruating girls (Jasper et al. 2012).

Additionally, WASH in schools programs can have a ripple effect – as children receive hygiene education in school they take the knowledge and practices home, becoming agents of change for their families and communities (Onyango-Ouma et al. 2005). The knowledge of WASH information does not necessarily translate into behavior change, particularly around more complex WASH activities like water purification, but education has been highlighted as a key place to begin (Freeman & Clasen 2011).

Without an enabling environment, including government oversight, funding, and institutionalization, WASH in schools programs will likely fail (Saboori et al. 2011). Many countries have policies regarding WASH in schools, but they are widely underfunded and sporadically enforced. If funding for WASH in schools programs exists, the budget line items are typically reserved for capital costs of building water and sanitation infrastructure and do not include on-going costs of operations and maintenance nor monitoring and evaluation (UNICEF 2011a). Without adequate funding and post-implementation government oversight of WASH in schools programs, the services are unlikely to be sustained over the long term. Similarly, means to resolve breakdowns in hardware and software must be part of the program design phase to ensure sustainability. Roles and responsibilities among all stakeholders especially governments, non-governmental organizations (NGOs), and communities are not often clearly defined which makes it difficult for post-implementation monitoring and evaluation (M&E) and to resolve breakdowns in the systems.

WASH in schools may fall under the purview of several government ministries depending upon the country, including the ministries of education, health, water, sanitation, public works, and rural development compounding the governance and funding issues. Diversity of government entities with full or partial oversight makes it difficult to establish and maintain WASH in schools programs. To overcome this confusion, many communities turn to local government officials and NGOs to address their specific WASH in schools-related issues (IRC 2007). While more effective in the short-term, concentrating efforts on local officials does not address systemic issues of a lack of national government funding or guidance on WASH in schools services.

Another challenge to sustainable WASH in schools programs is institutional acceptance, use, and maintenance of the WASH facilities. The presence of toilets, for example, is not synonymous with using and maintaining those facilities. If teachers and administrators are not trained and incentivized to follow national WASH in schools policies then the programs can jeopardize the intended sustainability, health impacts, and learning outcomes (UNICEF 2011a). Similarly, educating local communities on WASH knowledge and practices is critical to institutionalizing WASH in schools programs (World Bank, Water and Sanitation Program [WSP] 2005). UNICEF's WASH in schools publication ‘Raising Even More Clean Hands: Advancing Learning, Health and Participation Through WASH in Schools’, highlights six points of action for stakeholders to improve the sector: (1) set minimum standards for WASH in schools; (2) monitor WASH in schools coverage through Education Management Information Systems; (3) engage with at scale WASH in schools programmes; (4) involve multiple stakeholders to support WASH in schools programmes; (5) contribute evidence on the impact of WASH in schools programmes; (6) raise the profile of WASH in schools programmes (UNICEF 2012).

Collection of valid and reliable data is one step toward effective M&E. The data must be analyzed and used to make informed program adjustments to fit the needs of the school body. Completing this M&E cycle ensures school staff, donors, and NGOs do not waste time and resources on WASH in schools hardware and software that will not endure beyond the implementation phase. Too often, monitoring focuses on the presence of facilities, not the functionality, quality, or adequacy to meet students' and teachers' needs. Evaluations are often tailored to funders, NGO board members and staff members, not host government ministries or school communities (Fine et al. 2000; UNICEF 2011).

WASH in schools programs are complex, requiring ongoing monitoring. The challenge is capturing enough valid and reliable information to make informed programmatic changes without unduly burdening school administration, or in some cases the surrounding community. While more NGOs are conducting WASH in schools programs globally, their M&E activities have not been studied. This study aimed to characterize how implementers and donors conduct M&E with the goal of identifying trends in the sector and providing recommendations to improve the effectiveness and sustained impact from these programs. Specifically, this study examined current M&E practices of WASH in schools programs, and the barriers implementing organizations face and overcome while implementing M&E plans.

Quantitative and qualitative surveys were conducted with 21 organizations – 17 international NGOs, 1 academic institution, and 3 donor organizations – as follow-up to the 1,000 School Initiative launched in 2008 to raise awareness and increase attention of the need for improved WASH conditions in schools. Over 40 implementing and donor organizations pledged to increase WASH coverage in 1,000 schools in 30 countries around the world and informally agreed to monitor these projects for three years post-implementation. From July 2012 to January 2013, 21 of the organizations consented to structured, in-depth interviews. Implementing and donor organizations were analyzed equally because of the high level of donor involvement with the programs. The surveys were not conducted with in-country government officials responsible for WASH in schools, which would have provided a valuable perspective.

Respondents were asked about their organization's M&E policy; how their collected M&E data influenced current or future programs; if they were aware of a national government monitoring system; and whether their M&E data were fed into such a system, if one exists. These questions sought to understand internal organization culture around M&E data uses and external integration into government systems to provide a sustainable framework for WASH in schools projects.

All of the interviews, except one, were recorded and transcribed; and transcripts were coded using grounded theory (Glaser & Strauss 1967). Letter identifiers were used throughout the paper for direct quotations and summary statements. One organization had three distinct programs and interviews; although under the same umbrella organization their responses are analyzed separately and identified as Organizations D1, D2, and D3. Respondent organizations implemented WASH in schools programs in 27 countries throughout Africa, the Americas, Eastern Mediterranean, South-East Asia, and the Western Pacific regions.

Fifty-seven percent of organizations reported both having an M&E policy and partially or fully implementing it on WASH in schools projects. Sixty-two percent of respondents reported their organization's M&E data always or sometimes influenced their future programmatic activities. Slightly over half (57%) of the respondents knew if a national government monitoring system existed or not in a project country; and only 19% of organizations said their M&E data interact with the government monitoring system in any way. Respondents also reported barriers to M&E, which are presented in major categories below in Table 1. In parentheses next to each barrier is the number of organizations mentioning that barrier.

Logistical barriers stemmed from issues related to collecting the M&E data, such as impassable roads to program sites or field staff being too busy to accurately collect data. In particular, five organizations recognized that their project sites were too geographically dispersed in a country or region to conduct M&E visits effectively. In response to this barrier two organizations mentioned they were restructuring their programs to ensure geographical proximity to facilitate more efficient, lower cost M&E activities. Four organizations working in Africa and the Americas mentioned the government's shuffling of teachers during the school year as a barrier. Organization B stated frustration at their M&E plan being derailed by the government moving a teacher recently trained in their program to a different school, bringing the local program to a standstill. Setbacks like these were experienced by several organizations and admittedly are extremely difficult to plan for in budgets and M&E timelines.

Three-quarters of organizations mentioned the lack of M&E capacity in the field. Organizations cited a general shortage of M&E capacity both internally and in their host government counterparts. Respondents said field staff did not have adequate M&E skills to design and execute an M&E plan that would yield quality data. People with M&E skills are often in demand, but especially in developing countries where the emphasis is on hiring local people who become even more desirable to other NGOs, bilateral and multilateral organizations, and businesses as their skills improve. It creates a serious brain drain for implementing NGOs who want to have in-country M&E staff but lose them to agencies that can pay higher salaries.

Within M&E capacity two headquarter office respondents said field staff did not appreciate the value of M&E, so it was not prioritized in their work plans. Organization Q said their challenge was ‘convincing the country office that they're not collecting data for us necessarily, but it is for them to look at, review, and use. [I]t's getting them to see evaluation as a useful tool rather than as a reporting tool.’

Funding for M&E was mentioned by over 50% of the respondents, including the general lack of sufficient funding to conduct effective M&E during and post-implementation and the need to focus on donor compliance rather than M&E plans written into programs. According to Organization B, donors are more concerned with programmatic outputs than the longevity or sustainability of a project. This meant insufficient M&E funding from the beginning or when an M&E line item does exist it is cut when programmatic costs unforeseeably rise. Without donors requiring them to conduct monitoring and post-implementation evaluation, NGOs necessarily moved on to the next funding source and program site. Even when organizations wanted to do post-implementation M&E to understand the long-term sustainability of a program, they often lacked funding. Organization D2 mentioned, ‘the budget we were allocated for this project didn't allow us to have any monitoring/evaluation human resource, which is [what] we often find in undertaking a project – there isn't funding for that position.’

Throughout the WASH in schools subsector, there are few widely accepted best practices and indicators to facilitate consistent M&E. Without generally accepted WASH in schools M&E indicators, individual NGOs and country offices created their own instruments that are influenced by the evaluator's personal interest and may not be more broadly applicable. The absence of standard metrics was compounded by the previously mentioned lack of M&E capacity among field staff. This led to an increase in headquarter staff responsibility to create M&E instruments that, while valid, may not be reliable within the local context or truly capture the data needed to inform future programming decisions.

Finally, five organizations mentioned the effect that working with in-country partnerships or systems had on their WASH in schools M&E. Several organizations mentioned that the inter-ministry nature of WASH in schools within government systems made M&E difficult. For example, the ministry responsibility for WASH in schools can change based on the location, whether it is a primary or a secondary school, or depending on which component of the WASH program (e.g., hygiene promotion, etc.) was implemented. Further complicating the matter is the lack of oversight and final decision-making authority or fiduciary responsibility for WASH in schools programs often given to one government ministry. Due to the lack of ownership and leadership by one government ministry, NGOs may serve as a convener, bridging that gap. As Organization E said: ‘[WASH in schools] is a difficult subsector as it intersects so many disciplines. Every time I've been at an event that brings the Ministry of Education and Water, and local governments together . . . it's never been a substantive discussion. It's taken so long to get people on the same page that it hasn't been productive.’ Additionally, two organizations spoke about the unique situation of schools within a community as hampering their M&E. For example, schools are surrounded by and interwoven into communities, and these communities may lack access to WASH themselves, so M&E efforts must take these factors into consideration.

As a follow-up question to barriers, respondents were asked how they were addressing the barriers they specifically mentioned; responses are summarized in Table 2.

When respondents were asked how they were addressing the barriers to M&E for their organization, the majority focused on challenges with socio-political barriers. Six organizations mentioned increasing their work with local and national government officials and with the community surrounding the schools as a way to institutionalize M&E. Organization P mentioned including as many local institutions as possible in their M&E data collection and analysis ‘so there's more resilience in the program.’ They also train parents, teachers, students, and community members on the WASH systems and the need for monitoring to ensure the systems last.

The next barrier most addressed was M&E capacity for organizational staff and government officials. When Organization L allocated more resources towards building their staff's capacity, they found that staff ‘want to do [M&E] because they want to see what happens in their program, but they are not trained,’ so providing some basic training mitigated that hurdle. Other organizations internally hired dedicated M&E staff or contracted the work externally to consultants. While this dedication of resources is important at the headquarter level, it can leave a handful of M&E staff members overseeing a huge portfolio of projects in multiple countries with little time to interact with the field staff or visit program sites.

When Organization N internally evaluated their M&E process, they realized that ‘we were collecting way too much data and it was overwhelming for people. [. . .]. We've really streamlined for the key things that we think are critical and that they agree are critical.’ Once they reduced the M&E data collection expectations and requirements of field staff, this organization found they received better results from the increased buy-in. With more training and without unnecessary M&E requirements, field staff also took a more active interest in ensuring data were collected, analyzed, and fed back into programs, successfully completing the M&E feedback loop.

Although 15 organizations said funding was an M&E barrier, only five provided specific examples of how they are addressing this barrier. Organization D1 placed more emphasis on educating donors to require M&E because ‘it's easier for us to influence our funders, rather than influencing all of the staff because the funders, to an extent, will drive all the programming anyway.’ Organization L commented that ‘as donors are more sophisticated and understand the value of [M&E] they are more willing to let you put it as a line item in the budget.’ Despite the clear funding challenges organizations face, few mentioned actively addressing this barrier. Another organization mentioned that M&E expenses are one of the first budget line items cut when programming costs unexpectedly rise. Additionally, several respondents felt the WASH in schools sector needed to clarify the NGO's responsibility in providing M&E after the formal conclusion of a WASH in schools program.

Finally, two organizations reported making programmatic changes to ease the burden of M&E on field staff. To address this issue, Organization O developed an internal rating system to evaluate the severity of a programmatic problem, making monitoring easier for field staff to rapidly assess the situation, but they are still in the process of implementing this solution.

The interviews revealed that organizations had made efforts to institutionalize WASH in schools M&E within national governments. These advocacy efforts are largely regarded as the long-term solution, but they also noted the sector must address more immediate sustainability concerns. The study participants reported taking interim steps to improve M&E while simultaneously working on strengthening government systems. One organization felt that governments must have a greater role in WASH in schools programming and M&E to ensure its sustainability in the future.

Overall, the trends seen in this study are encouraging. Organizations implementing WASH in school programs are rethinking how they conduct M&E. As donors become more convinced of the importance of M&E for sustainability, these organizations can share their M&E challenges and receive the financial and informational resources necessary to overcome barriers.

The WASH in schools sector needs to discuss these issues openly with the education sector, without fear of reprisal, to foster sustainable activities that provide students with a safe and hygienic place to maximize their learning potential. It is time for increased sector-wide learning about failures, as well as successes, and moves toward more sustainable WASH in schools programs. The WASH in schools sector struggles to close the M&E cycle of resolving issues identified through both ongoing monitoring and long-term evaluation of a project and misses the opportunity to feed the collected data back into programs. As Table 1 shows, identified M&E barriers are not actively addressed by WASH in schools organizations.

Although 71% of organizations mentioned logistics as an impediment to M&E efforts, only 9.5% reported making changes to improve this issue. Conversely, socio-political barriers around working with governments and communities was mentioned by 19% of organizations as an M&E barrier, but 48% of respondents reported focusing on improving these relationships to help their M&E processes. The socio-political barriers category may present the most positive finding of the study because organizations may be adequately focusing on this M&E hurdle. Respondents may not have identified socio-political dynamics as a greater barrier because organizations have already strengthened those relationships, but this area was not explored in the interviews. Over half of the respondents said M&E funding was an issue, but 19% mentioned actively addressing that barrier within their organization. The only category where a barrier was addressed almost as often as mentioned was with M&E systems. Thirty-three percent of respondents said the lack of M&E systems was a barrier and 23% of the same respondents said they were actively streamlining M&E expectations and demands upon field staff to overcome this barrier.

These trends do not imply WASH in schools implementers are ignoring best practices within the sector. In fact, three of UNICEF's Six Points of Action mention the need to institutionalize WASH in schools M&E into existing government systems, like the Educational Information Management System or Health Management Information System, and to work with local and national government officials to raise awareness for WASH in schools. Advocating for national monitoring systems and navigating government bureaucracies is a slow process, but it is the most effective way to make WASH in schools institutionalized and sustainable. Despite the nearly 50% of respondents that said working closer with governments is a priority, only four organizations said that their data interact with national systems in any capacity. Only about half of those surveyed knew for certain whether a national monitoring system existed. It is possible that systems may not exist in the countries where respondents work.

Since organizational data collection rarely aligns with government systems, parallel M&E systems may exist that are collecting similar data, duplicating efforts and resources needed to collect, analyze, and act upon findings. Without a solid government monitoring system in place, organizations cannot be assured that their WASH in schools efforts will continue beyond the life of a funding cycle or their presence in a country. In this situation, post-implementation M&E by the organization is key to program sustainability, as they could provide M&E support while a government system is established or while WASH in schools M&E is included in existing government monitoring systems. Currently, few governments and NGOs have the financial or personnel capacity to conduct post-implementation M&E, but this issue was not explored in this study. Funding for M&E during and post-implementation is another issue that arose from the interviews. During implementation, organizations are reliant upon construction, transportation, and personnel costs remaining static to preserve an M&E budget line item, but unforeseen factors often disrupt proposed budgets. Donors have significant power over how programs are implemented and evaluated, and the NGOs respond to their requests.

Given the lack of attention around M&E capacity in organizations and within governments, it is not surprising field staff do not understand how critical M&E is towards providing sustainable services. Organization E mentioned that the sector leapt at WASH in schools M&E without understanding why it was necessary and without the capacity to collect valid information to inform future programs. Encouragingly, respondents indicated NGOs are moving away from haphazard M&E activities and are moving towards more streamlined and institutionalized systems to improve program sustainability. Identifying examples of how organizations and governments have integrated their M&E systems collaboratively will be useful as fledgling programs develop.

Although monitoring and evaluation are distinct activities, this study refers to M&E collectively following the lead of respondents when discussing their programs. Where possible, post-implementation M&E was pulled out and distinguished from implementation-phase M&E activities. Selection bias was possible because respondents self-selected through participation in the 1,000 Schools Initiative without randomization. Respondents are not necessarily representative of all WASH in schools organizations, but the sample includes organizations of diverse sizes, budgets, scopes, and backgrounds. Another limitation is the lack of government officials in the sample. As the sector looks to institutionalize WASH in schools M&E within relevant government ministries, interviews with government officials in charge of national monitoring systems or WASH in schools programming would be extremely valuable. Future research is needed to better characterize the roles and responsibilities of organizations in relation to government stakeholders and how M&E, and the resolution of problems, can increasingly become part of the national and local governments’ role in operating and maintaining WASH in schools. There is also a need to understand the role, and M&E practices, of private companies that are increasingly conducting WASH in schools as part of corporate social responsibility.

Based on this study, we developed five recommendations. First, there is a need for organizations to conduct internal assessments of their M&E barriers, policies, and practices. Several organizations mentioned starting this process but few had completed it. In some situations, these conversations improved M&E and helped organizations to identify and address their unique barriers at headquarters and in the field. Second, organizations implementing WASH in school programs need to determine how long-term M&E activities will take place after their funding ends. Organizations will need to work more closely with governments to understand what M&E efforts are feasible on a long-term, continuing basis by the government and how the organizations can support those efforts. This is imperative if long-term impacts are desired. This study found that while M&E is planned for in all WASH in schools programs, unforeseen issues or budget changes often hinder those good intentions. The Dutch government's aid ministry has included a sustainability clause into their WASH projects (Moriarty 2012). This clause will provide funding for and require all WASH programs to adhere to an agreed-upon level of service for 10 years following implementation (Sanitation & Water For All Statement 2012 – Government of the Netherlands 2012). The United States Agency for International Development (USAID) is following suit in their Water and Development Strategy 2013–2018. The agency lists ‘investments in longer-term monitoring and evaluation of its water activities in order to assess sustainability beyond the typical USAID Program Cycle and to enable reasonable support to issues that arise subsequent to postcompletion of project implementation’ as a priority (USAID 2013). It remains unclear, however, how NGOs can efficiently and effectively shift over to an enduring system of service and delivery that governments should ideally provide. Third, organizations need to educate all stakeholders on the importance of M&E and be more transparent with their findings. It is also important that M&E is developed with a longer term perspective that will likely involve national and local governments managing the monitoring system. Donors have the opportunity to facilitate this process. As Organization A said, ‘Part of the culture of NGOs [is] that their focus is mainly on donor compliance and the objectives and then reporting, so you focus on that more than the wider picture.’ A stronger commitment to M&E and an effort to integrate the monitoring system into an enduring national system will help ensure that the money and time spent on data collection, analysis, and issue resolution has a lasting impact. Local and national government officials and community members must also see the value of collecting and using M&E data to improve the sustainability of their programs. Fourth, there is a need to create standard indicators to facilitate M&E activities. M&E is more difficult for smaller NGOs or those without dedicated M&E staff in the field, and the availability of tested, reliable, and valid metrics on critical indicators would reduce the resources needed to conduct M&E and produce comparable results across programs. Three organizations explicitly mentioned streamlining M&E instruments across all WASH in schools programs. Organization N worked with field staff to create feasible and reasonable M&E tools that did not overburden already busy field staff. They found in the field staff consultation process that the ‘key thing is really [to] ensure that [M&E data] is coming back to the partners on a regular basis, so it's not just this data that nobody ever sees again.’ Standardized indicators would make such a process faster for organizations and improve M&E within the sector. Finally, there is a need to clarify roles and responsibilities for post-implementation M&E. Implementing organizations need to clarify with international donors, country governments, and communities their responsibility to conduct post-implementation M&E. This recommendation requires a conversation among WASH in schools stakeholders to determine realistic expectations for post-implementation M&E and how donor-funded programs can fit into the local context and have a long-term impact on the health and learning of children. The experiences from the WASH in schools sector will likely be useful in other settings and programs – WASH in healthcare facilities, for example – and more work will be needed to transfer the lessons learned as efforts to scale up WASH in all public spaces occurs.