Engaging with the WASH enabling environment

Approaches and thinking in the WASH sector have evolved over the past 30 years as implementers and practitioners have grappled with two fundamental issues: efficacy and sustainability.

The former issue has been addressed through increasing integration of health and environmental considerations – including a contemporary focus on behaviour change. This aspect of the sector evolution is witnessed in the change from a predominant emphasis on clean water supply during the international drinking water decade (1981–1990) (http://www.gdrc.org/uem/water/decade_05-15/first-decade.html); through an emphasis on water and sanitation (‘WatSan’) supply as articulated in the Millennium Development Goals (Target 7C: Halve, by 2015, the proportion of the population without sustainable access to safe drinking water and basic sanitation (https://www.un.org/millenniumgoals/)); to the emphasis on water, sanitation and hygiene (WASH) integration (Scott et al. 2003), and now safely managed WASH under the Sustainable Development Goals (SDGs).

The latter issue remains problematic for the sector. Whilst the links between WASH and health have been well established (WHO 2008), the factors that enable the improved health outcomes to be sustained into the future are less well understood. This issue affects both the infrastructure provided during projects, and the improved behaviours that are fostered – such as using toilets, washing hands and so forth. Studies and project evaluations have shown that both infrastructure and behaviour decline once donor-led WASH interventions end (Pickering et al. 2019). Indeed, much of the work done by the WASH sector has been described as ‘islands of success in a sea of failure’ (Schouten et al. 2003).

Recognising this challenge, contemporary approaches in the sector have sought to re-frame WASH away from a traditional community development model, instead situating it within sub-national governance systems. That is, traditionally WASH has been seen as a community responsibility (Schweitzer & Mihelcic 2012), and WASH programs have sought to strengthen local community structures and empower members with the skills and resources required to maintain and manage WASH infrastructure and sustain improved hygiene behaviours. Critique of this approach has recognised that not only do communities need to be connected with the outside world through supply chains for goods and services, and to wider health systems (WaterAid 2011 Sustainability Framework), but also that bestowing sole responsibility for the ongoing management of water and sanitation on communities unfairly absolves government of its responsibility to deliver these basic human services.

This paper presents the experience and lessons arising from a long-running global WASH program – the Civil Society Water, Sanitation and Hygiene (CS WASH) Fund (An AUD103 million investment by Australia's Department of Foreign Affairs and Trade (DFAT) that spanned March 2013 to October 2018) — that overtly sought to tackle the fundamental issue of WASH sustainability by discouraging the direct delivery of services by the civil society organisations (CSOs) funded by the program, instead focussing attention and effort on the connections between communities and the broader environment in which they are situated, referred to in this context as the WASH Enabling Environment (UNICEF 2016).

Specifically, three tools are presented, that were developed to prompt internal debate and reflection amongst project implementing teams. The tools were developed by a three-member monitoring, evaluation and review panel (MERP) engaged by DFAT for the life of the Fund to support the performance of the projects in the Fund.

The CS WASH Fund comprised 29 WASH projects implemented by 13 CSOs in 19 countries throughout Africa, South Asia, South East Asia and the Pacific. In recognition of the limited sustainability of an earlier phase of WASH programming, the Fund was overtly designed (https://dfat.gov.au/about-us/publications/Pages/civil-society-water-sanitation-and-hygiene-wash-fund-design-document.aspx) to prompt greater engagement with the enabling environment by CSOs, moving away from the traditional direct delivery approach whereby project teams worked directly with beneficiary communities and with project scope framed by donor-funded deliverables.

The Fund promulgated an ‘actor-based’ theory of change (ToC) that highlighted the interaction of project delivery teams (CSO Project staff and also any others directly contracted to provide services) with groups of people – collectively referred to as change agents – who have ongoing responsibility for delivery of WASH services in communities. Each project defined and identified the change agents with whom they would engage, and by-and-large these comprised government staff at local, regional and national levels, private sector actors and to a lesser extent members or staff of non-government organisations. Projects were obliged to seek ways to strengthen and support these people such that their work delivering WASH services could benefit a target population in perpetuity. Thus, in the Fund's vernacular, project staff (delivery teams) implemented activities to produce project deliverables designed to strengthen and support the work of change agents, whose performance changes were defined as expected changes. The expected changes, if realised, were hypothesised to foster improved WASH services for the target population – defined as the project's impacts. Hence, the Fund's overall ToC (illustrated in Figure 1) was that through targeted interventions, CSO delivery teams could enhance the performance of various WASH sector change agents, thereby improving the enabling environment for sustainable WASH services in communities, and thus fostering sustainable environmental health outcomes beyond the life of donor assistance.

This relatively simple ToC provided a framework that united 29 quite disparate projects, each with their own contextualised project ToC. Inherent in the term enabling environment is a complexity that can potentially encompass all aspects of socio-economic development (Brinkerhoff 2004) much of which was reflected in the diversity of contexts across the 19 countries and four regions of the Fund. However, the fund ToC was premised on the straightforward notion that all development activities, regardless of the context, involve humans interacting to foster desirable changes. This fundamental reality provided a unifying narrative for the 29 project teams, focussing effort and resources on identifying key WASH change agents and nurturing their improved performance.

Two projects in PNG illustrate how this worked. One identified the lack of a national WASH policy as a significant bottleneck to the provision of sustainable WASH services in their target Province (East Sepik), and so sought to work with and support key change agents not only at Province and District levels, but also in the national government tasked with developing the policy. Their project-level ToC was that their work in communities in East Sepik would be more sustainable with the policy and associated regulatory framework and financing mechanisms in place. Another project was working in a Province where government service delivery had completely broken down (Western Province). In the absence of effective Government change agents, they identified change agents at community level – community leaders, school teachers and so on – as the key to sustainable WASH services in these remote communities and so focussed their efforts on supporting these people. These two projects, operating in one country, had very different ToCs based on their operating context, but both were captured by the Fund's overarching ToC.

This paper describes three tools developed by the MERP and applied systematically throughout the Fund to measure engagement with the enabling environment and the likely sustainability of project impacts:

• Strategy Mapping Tool: a process to prompt critical reflection about the effectiveness of approaches employed to improve WASH services and the extent to which these strengthen or engage with the enabling environment.

• Context Mapping Tool: a process to prompt critical reflection about the relevance of project approaches within the operating environment, and the extent to which engagement with the enabling environment is possible or feasible.

• Change Agent Assessment Tool: a process to prompt critical reflection about the likely sustainability prospects of WASH sector change agents' performance as a result of the project's efforts.

The paper concludes by presenting preliminary analysis that appears to support the hypothesis that greater engagement with the enabling environment does indeed foster greater sustainability of WASH services. But first, the following sections describe each of the three tools.

Initially, the MERP developed the Strategy Mapping Tool as a way to analyse and characterise the breadth of approaches employed by 13 CSOs in 19 countries implementing 29 WASH projects. It became evident that such analysis provided a powerful prompt for reflection and discussion within and between project delivery teams about the relative merits of project interventions and approaches.

The basis for the ‘strategy map’ was a categorisation of the WASH approaches articulated in the 29 project operational plans (comprising sub-plans for integrated WASH: water supply, sanitation, hygiene promotion, gender and social inclusion (GESI) and environment, climate change and disaster risk reduction (ECD)). The types of ‘strategies’ employed by CSOs were conceptualised along a continuum or spectrum: from little engagement with the enabling environment (i.e. direct delivery) at one end through to full engagement (i.e. strengthening change agents in the enabling environment) at the other (Figure 2).

Implementing the strategy mapping tool involved participatory discussions with CSO project delivery teams (strategy mapping was facilitated by the MERP during ‘learning events’ convened by the fund) to map each project deliverable against the spectrum (typically, each project defined between 30 and 50 deliverables across the five sub-plans). The discussions were facilitated by the MERP to promote constructive reflection and ensure the integrity of the mapping process. An Excel spreadsheet was developed to give structure to the discussion, automate the processing of the data, and display a ‘strategy map’ for each project. The structured process strengthened the integrity of the maps by minimising the potential for participants to pre-empt the final map and ‘game’ the scoring during the discussions. A typical data entry sheet is shown in Figure 3.

In addition to mapping the deliverables onto the strategy spectrum, delivery teams also mapped them to established WASH sector intervention domains: infrastructure, behaviour change, GeSI, ECD and Policy/Governance.

In practice, the mapping process involved debating where on the strategy spectrum each deliverable best sat; under which intervention domain; and then entering group decisions into the spreadsheet tool. The scoring algorithm aggregated the individual scores (scores were from 1 (direct delivery) to 5 (strengthening the enabling environment)) and presented this as a proportion of the maximum possible score for each domain. Scores were then displayed as a radar chart – the ‘strategy map’ (Figure 4). The tool also enabled disaggregation of results by change agent type: government, private sector or civil society.

The strategy mapping tool was developed primarily as a structured reflection tool for project delivery teams. It enabled an appraisal of the extent to which projects were aligning with the Fund's ToC, and identified areas of strength and weakness, in turn prompting adjustments in programming. For example, the map in Figure 4 shows a general weakness in the ECD domain, but a commensurate strength in behaviour change.

Project delivery team members reported that whilst they had a general sense of the strengths and weaknesses in their approaches, the strategy mapping exercise highlighted issues for attention in a systematic way. The visual ‘map’ was considered helpful for triggering discussions about project improvement.

Notwithstanding the reported benefits of the strategy mapping tool discussed above, a valid criticism was its limitations when comparing projects across the Fund due to the diversity of operating contexts. That is, while the strategy maps highlighted the alignment of individual projects with the Fund ToC, they did not reflect tactical decisions made by delivery teams to accommodate contextual considerations. For example, projects in Zimbabwe and in the Western Province of PNG had very different strategy maps, but for good reason. In Zimbabwe, two projects actively engaged local government change agents who were amenable to influence due to a legacy of well-functioning local government that had lapsed during periods of national crisis. Whereas in the Western Province of PNG, there was virtually no previous or current local government or private sector activity. As a consequence, project delivery teams had no choice but to implement directly within communities. When set alongside the Zimbabwe project strategy maps, the PNG Western Province map gave the impression that this project was poorly aligned with the Fund's overall ToC.

In response to this observation, and to prompt meaningful Fund-wide discussion and learning, the MERP developed a context mapping tool that was implemented in conjunction with the strategy mapping. The structure of the tool borrowed heavily from a similar tool (https://www.ircwash.org/news/wash-sustainability-assessment-tool) developed by AguaConsult (https://www.aguaconsult.co.uk) as part of the IRC's Triple S Program but was dramatically simplified and tailored to complement the strategy mapping tool and the Fund's operational plan components.

As with the strategy mapping tool, the context mapping tool involved participatory discussions with project delivery teams facilitated by the MERP during learning events; however, the context mapping also included project change agents. The process involved a detailed discussion concerning the status of each of five ‘context domains’. Participants rated these domains to arrive at a score out of 100 representing the extent to which the context was amenable for engagement by the project. (The rating process was made systematic by elaborating each of the five domains with ‘sub-domains’ that were presented as a series of descriptions/scenarios arranged along an ordinal scale. Participants simply debated and selected the scenario that best matched their observed experience within each domain to generate a ‘score’. For example: a sub-domain of ‘Infrastructure’ was ‘Government involvement in the delivery of WASH infrastructure’. Projects operating in the Western Province of PNG beyond the reach of government services scored this zero. Whereas, projects operating in Zimbabwe with a functioning but struggling government structure awarded a higher score. A typical data entry page for a sub-domain (in this case ‘Government involvement in the delivery of WASH infrastructure’) is shown in Figure 5.)

The strength of the context mapping process derived largely from robust discussions between project delivery teams and the change agents required to arrive at an agreed score for each sub-domain. The contestability in these discussions produced constructive insights and useful maps.

The final step involved the tool displaying a radar chart with the strategy map (Figure 4) superimposed over the context map. This was to draw focus on how well aligned – or relevant – the selected WASH strategies were within a project's operating context, as illustrated in Figure 6.

The example in Figure 6 shows a case in which the project strategies (black line) were ahead of the operating context (green shape) in all domains except ECD. One interpretation of this could be that the project's engagement with the enabling environment is strong in all the domains except the ECD domain where it is weak or tokenistic. The merits of this situation could then be debated with questions such as: (i) to what extent is the project leading the context too much in relation to four of the strategy domains – with a consequent risk of unproductive reform and project failure?; (ii) to what extent is the project lagging the context too much in relation to ECD, and failing to achieve reforms that are otherwise possible? (iii) to what extent are the strategy choices simply pragmatic and reflect what is possible in the operating context or within the capacity of the delivery team?

The parallel application of the strategy mapping and context mapping tools generated fruitful debate that frequently resulted in amendments to project approaches or emphases. To the extent that these discussions involved people with influence and with knowledge of the projects and contexts, the tools contributed to more successful projects, and stronger WASH outcomes.

Although both tools relied on the subjective judgements of project stakeholders, the validity of the results was underpinned by several features of the processes: (i) the fine granularity of the scoring processes (i.e. many small judgements abstracted from the implications, and then aggregated); (ii) external facilitation of the process by people familiar with the projects and tools (i.e. the MERP) who were able to challenge unrealistic scores; and (iii) the contestable and transparent discussions that led to the scoring. In general project teams agreed that the maps were representative of their project approaches and the contexts in which they were being implemented.

The third tool was rolled out during the second half of the Fund implementation period to gauge the likely sustainability of project impacts. According to the Fund's ToC, the sustainability of impacts is underpinned by the sustainability of improvements in the performance of change agents to deliver WASH services. Hence, the MERP reasoned that focussing on these improvements (i.e. the ‘upstream’ expected changes) would yield insight into the likely sustainability of the WASH benefits in target populations (i.e. the ‘downstream impacts’). This was done systematically through the application of the Change Agent Assessment Tool (CAAT).

There is an inherent tension in the Fund's ToC between impact measured simply by the numbers of people with access to – or use of – WASH services, and the sustainability of that impact driven by a strengthened enabling environment. Put simply, a project may achieve impressive impact metrics by implementing directly in target communities, but experience suggests these impacts are unlikely to endure (UNICEF 2016) (Concept Note: Towards Sustainable WASH: Strengthening WASH Governance through Accountability. Prepared by the UNDP Water Governance Facility, Stockholm International Water Institute, September 2014. Available at: https://www.unicef.org/wash/files/Accountability_in_WASH_Explaining_the_Concept.pdf). Conversely, a project that works to strengthen the enabling environment for WASH service delivery may generate less impressive impact metrics in the near term but is arguably more likely to foster enduring change.

As with the other tools, implementing the CAAT involved a participatory workshop with project delivery teams facilitated by the MERP (implemented during on-site project monitoring visits rather than at fund learning events). Participants reflected on three stages of project support for sustainable WASH services: (i) the past – the extent to which the project had completed the planned deliverables in relation to each class of change agent; (ii) the present – the extent to which the change agents were performing to expectation at the time of the workshop; (iii) the future – the extent to which change agent performance was considered likely to endure beyond the project.

The first step involved mapping the change agents to the relevant deliverables in the operational plans. The extent to which the project had completed planned deliverables produced a score that reflected the proportion of total effort invested in each class of change agent (N.B. other change agent characteristics were also recorded, such as: the number, gender, type (government, private sector, community), relative importance to the project ToC, and the level at which the change agents were operating (national, sub-national or community)).

The second step involved discussing in detail the performance of each class of change agent relative to expectations (at the time of the CAAT workshop), as articulated in the project operational plans. Project teams rated progress towards expected performance for each class of change agent against an ordinal scale shown Table 1 (Performance Now column).

The third step involved debating the likelihood that change agent performance would be sustained beyond the life of the project. This discussion was made more systematic by having delivery teams score sustainability with reference to a five-element conceptual model of sustainability shown below (Table 2). (Arguably, each of the five elements is a necessary but not sufficient condition for sustainable WASH services. The implication is that all five elements must be in place for sustainability to be optimized.) Again, delivery teams rated each of the five elements against a three-point ordinal scale for each class of change agent (Projected Performance column in Table 1).

The sustainability score was the aggregate of the scores against these five elements. Teams were also asked to provide justifications for the scores, including evidence where possible.

The tool (deployed in Excel) assimilated workshop data into a single graphical representation of change agent performance and sustainability – a bubble chart comprising four dimensions. The example in Figure 7 shows a project working with seven classes of change agent, represented by seven bubbles. The size of the bubbles reflects the extent to which change agents are performing to expectation (large being all on track or better, small being minimal progress). The colour conveys the extent to which the project deliverables relating to that change agent are complete (green being completed and red being not started). The Y-axis is the level at which the change agent operates within the national administrative structure; and the X-axis is the likely sustainability of the performance gains. Ideally, towards the end of the implementation period, one would see large green bubbles clustered over towards the right-hand side of the chart. Conversely, small, green bubbles towards the left-hand side of the chart indicate sustainability issues as the project's deliverables are complete and yet the expected performance gains have not been realised and sustainability prospects are limited (small red bubbles are also problematic but reflect a situation in which the project has not yet produced the planned deliverables – and hence issues may be more of a management nature rather than a development or context nature).

As with the other tools described above, the power of the CAAT was in the discussions and the opportunity this allowed for reflection on areas of weakness or strength in the project approach. When the workshop process was undertaken with sufficient implementation time remaining (for most projects the CAAT was applied with at least 6 months of implementation remaining), the output informed management decisions concerning resource allocation and effort. Also aligned with the other tools, the validity of the CAAT results was underpinned by the focus on detail when scoring without a view of the big picture, and again, the project teams endorsed the representation of their projects.

The imperative to develop and apply these tools primarily arose from the wider knowledge and learning agenda for the Fund. That is, the aim was to stimulate discussion and reflection amongst the project delivery teams concerning how effectively their projects were aligning with the Fund's overall ToC in relation to strengthening the WASH enabling environment. However, the systematic application of the tools across the project portfolio also produced three related datasets that were used to assess the Fund's ToC. (The context mapping tool was developed after the strategy mapping tool, and the timing meant that three projects did not develop strategy maps. As a result only 26 of the 29 projects are included in this analysis.) That is, correlations between the data captured by the strategy mapping, context mapping and CAAT tools was used to test the hypothesis that strengthening change agent performance would indeed yield more sustainable improvements to WASH services.

Figure 8 presents the average strategy scores for all domains set against the sustainability scores, revealing a positive but very weak correlation. However, if the strategy scores are moderated by the context scores by subtracting the difference between the two, thereby preferencing not just greater engagement with the enabling environment, but alignment with the operating context as well, the strength of the correlation increases (Figure 9). Further research could be designed to more precisely test the positive correlation, noting the subjective nature of the scoring process. Nevertheless, as discussed above, the subjectivity is moderated somewhat by the level of aggregation in the process (many small decisions/scores being made to contribute to the overall score) along with the fact that the discussions were moderated by the MERP, providing a level of consistency to the process. Further, there is a high degree of consistency in correlations using various disaggregations of the overall dataset. For example, a similar correlation is seen when the data is disaggregated by WASH sector domain (infrastructure, behaviour change etc.), and by change agent type (government, private sector, community).

The tools and data presented here are the result of the collective efforts of the 29 project teams that comprised DFAT's CS WASH Fund from 2012 to 2018. The tools were developed by the MERP and applied systematically across the Fund for both structured reflection purposes and for evaluating the validity of the Fund's overall ToC. Arguably this was only possible because of the latitude and relative freedom given to the MERP by DFAT, and the close, collaborative working relationships developed between the MERP and the CSO project teams.

The development and application of the tools presented in this paper generated datasets that could be combined to test the hypothesis behind the Fund's ToC: that greater engagement with the enabling environment would increase the sustainability of the Fund's goal and objectives. With the caveat that the findings were dependent on a degree of subjectivity within project teams, and that the likely sustainability is a projection rather than a measurement of actual future performance, the analysis cautiously indicates a positive trend in support of the Fund ToC. This would seem to justify the sector policy decision to shift investment towards strengthening the enabling environment for WASH; and away from direct investment approaches. Resourcing an ex-poste evaluation of the Fund to re-apply the tools after several years could verify the preliminary conclusion and ascertain the extent to which improved WASH services arising from DFAT's AUD103 million investment have been sustained.

Furthermore, while the tools were designed to dovetail into the Fund's operational plans, and to align with the Fund's ToC; they also have the potential to be adapted to any development initiative that aims to engage with the enabling environment, and can support an assessment of alignment between any project approach and the operating context. Hence, as well as being useful tools for evaluating projects and programs, they also have potential as management tools to check the logic behind project designs, and to track progress towards outcomes.

The authors would like to acknowledge the contribution of the Fund Management Team to both the development of the tools and their application — particularly that of the Fund's Knowledge and Learning Manager, Bronwyn Powell, and the Fund Manager, Amanda Morgen. The enthusiastic engagement of the CSO teams and their change agents is also acknowledged, as is the support to the entire process provided by DFAT.