Individuals working within the water, sanitation and hygiene for development (WASH) sector grapple daily with complex technical, social, economic, and environmental issues that often produce unexpected outcomes that are difficult to plan for and resolve. Here we propose a method we are calling the ‘Participatory Systems-based Planning and Evaluation Process’ (PS-PEP) that combines structural factor analysis and collaborative modeling to guide teams of practitioners, researchers, and other stakeholders through a process of modeling and interpreting how factors systemically and dynamically influence sustained access to WASH services. The use and utility of the PS-PEP is demonstrated with a regional team of water committee members in the municipality of Jalapa, Nicaragua who participated in a two-day modeling workshop. Water committee members left the workshop with a clear set of action items for water service planning and management in Jalapa, informed by the analysis of systemic influences and dependencies between key service factors. In so doing, we find that the PS-PEP provides a powerful tool for WASH project or program planning, evaluation, management and policy, the continued use of which could offer unprecedented growth in understanding of WASH service complexity for a broad spectrum of service contexts.
There is growing consensus within the international community development sector that development programs are interventions into complex adaptive systems (CASs) (Neely 2015a; Amadei 2016). While CASs can be defined as networks that are interdependent, emergent and path dependent (Lyons 2004), there is little agreement on what this means for development policy or practice. In the water, sanitation and hygiene (WASH) sector, the combination of social, political, technical and environmental factors ensures that interventions are themselves CASs that frequently act in confounding ways upon communities that are also themselves CASs (Neely 2015b).
Understanding WASH as a CAS which is interdependent, indicates that community WASH programs will intersect with other ‘systems’ – the socio-political, the environmental, and the organizational. Changes in these interacting systems have the potential to impact at the local level, causing desired changes, or undesirable effects such as policy resistance (Cairney 2012). Local systems influence each other, and local improvements can be shared and adapted across regions through social and professional networks. WASH programs also create changes in communities and can therefore cause local emergence; the patterns that form from new interactions are the basis for the emergence of new community organizations, relationships or external interventions. Emergence of grass roots organizations bodes well for increased traction of development outcomes, as they can be seen as an indicator of increased socio-political complexity and development (Neely 2015b). Problematically, path dependence in CASs means that slight differences in initial conditions across communities can result in WASH projects and programs evolving in ways that are difficult to plan for (Neely & Walters 2016).
Numerous studies have focused on the development and application of tools to ascertain factors that play a role in sustaining rural WASH services in developing countries. Some have approached this topic using composite scoring of aggregated factors to determine and assess sustainability with regards to some sustainability ‘threshold’. Examples include the ‘sustainability snapshot’ by Sugden (2001) and the ‘sustainability check’ by Godfrey et al. (2009). Other techniques are based on multivariate (Narayan-Parker 1995; Sara & Katz 2004; Foster & Hope 2016), and logistic regression analyses (Foster 2013). Some methods explicitly treat the complexity of WASH interventions with systems approaches, such as lifecycle assessment and integrated water service management (Xue et al. 2015), agent-based modeling (Mellor et al. 2012), Bayesian-networks (Fisher et al. 2015), multi-criteria analysis (Panthi & Bhattarai 2008) and probabilistic graphical modeling (Walters & Chinowsky 2016).
Each of these methods, techniques, and tools have advantages and disadvantages. Composite scoring methods allow researchers to assess sustainability without requiring hard data to support research findings, yet lack the unbiased rigor of statistical-based techniques (Lockwood et al. 2003). Regression-based methods are able to surmount or minimize biases, yet often require datasets that are either unavailable or prohibitively expensive (Lockwood et al. 2003). Neither composite nor regression analysis are able to express systematic relationships and indirect influences between factors (Jordan et al. 2011; Walters & Chinowsky 2016). Systems-based approaches are able to understand both direct and indirect influences, yet have weaknesses related to data collection, such as dataset biases and availability (Starkl et al. 2013; Fisher et al. 2015). The approach taken within the present study integrates the analytical power of statistical and systems approaches with the inclusive participatory practice of group model building, an approach the authors are calling the ‘Participatory Systems-based Planning and Evaluation Process’ (PS-PEP).
Historically, participatory approaches to WASH have included: community-led total sanitation (CLTS) (Kar et al. 2008), participatory hygiene and sanitation transformation series (PHAST) (Sawyer et al. 1998), self-esteem, associative strength, resourcefulness, action planning, responsibility (SARAR) (Srinivasan 1993), methodology for participatory assessments (MPA) (Dayal et al. 2000), and participatory rural appraisal (PRA) (Chambers 1994). Most participatory approaches are applied only at a community level and rely on variations of peer pressure (CLTS), community mapping, and seasonal calendars and transect walks (PRA, SARAR, PHAST, CLTS). The MPA is designed as a tool that can be used at different levels from community to policy, and with different purposes, but it is a scoring tool based on a preconceived set of values around water services. The PS-PEP presented herein builds on a tradition of participatory workshops and co-production of knowledge that has been good practice in community development over the last 30 years. While the focus of this paper is to explain the process and the tool, it should be noted that the process requires a facilitator who is familiar with systems thinking and the PS-PEP tools and has the skills to negotiate power and conflict within a small group.
Very few approaches to WASH service planning exist that combine participatory and systems-based approaches, although partial exceptions do exist within international development (e.g. Hovmand et al. 2010; Hovmand 2014). The study presented here seeks to address the limitations identified for other methods by developing an integrated tool within a participatory systems-based process for WASH program planning and evaluation. We propose the PS-PEP technique as a means to apply structural factor analysis within a participative workshop environment to elucidate the complex interdependent barriers and drivers of sustainable WASH services. The PS-PEP process is intended to aid WASH practitioners and policy makers as a decision support tool for regional planning and management of rural WASH. Information and experience is aggregated from across a regional area so that patterns of either sustainability or WASH service failure are shared and compared to elucidate significant factors. The regional view allows the boundary of the system of interest to be drawn more broadly than in community participatory approaches. Highly context specific, the regional boundaries considered by PS-PEP stakeholders are selected based on the primary geo-political influences on the WASH service of interest. The regional level of application of the PS-PEP approach means that it can incorporate political action, regional water basin management, land use and regional community associations. Participants for the workshops may include community members and local and governmental agencies, as the modeling aspects are adaptable to a range of learning and engagement styles. The benefit of the PS-PEP is to enable planning and management teams at all levels to achieve a more complete understanding of WASH service complexity. The models produced through the workshops highlight contextually important or impactful areas for allocation of resources.
In the sections that follow, we formally introduce the PS-PEP and demonstrate its utility with a case study in Jalapa, Nicaragua, in which a group of seven water committee members were guided through the process of model building and interpretation. In this case study, the region is based on the municipality (Jalapa), as this is the governance level that is perceived to have the most influence on both local water supply infrastructure and water basin management.
In analyzing the benefit of engaging Jalapa water committee members within the PS-PEP, we specifically seek to answer the following research questions:
What do the participants learn through the modeling process?
Does participants' learning demonstrate a systems-based understanding of water service complexity?
How does this knowledge influence their strategic planning and management of current and future water services?
By addressing these three research questions, this study aims to contribute a useful technique for the WASH sector that can be employed and expanded upon to bolster theory and practice related to sustainable WASH service planning, evaluation, management and policy.
METHODS AND RESULTS
At its core, the PS-PEP builds on participatory development and group model building approaches via the MICMAC (matrix of cross impact multiplications applied to classification) method – a structural factor analysis technique that entails the creation, manipulation and analysis of impact matrices to infer factor importance and evolution. Impact matrices house information related to the presence and strength of influence or dependency between factors thought to cause a particular outcome (i.e. long-term water supply infrastructure functionality). A common scale used to represent influence strengths is 1, weak; 2, moderate; and 3, strong (Godet 1976; 1986; Godet & Roubelat 1996; Scholz & Tietje 2002; Arcade et al. 2009). By itself, an impact matrix with weighted influence scores represents direct connections between factors. Iterative manipulation and reassessment of the impact matrix (MICMAC) can be used to infer indirect relationships in the form of hierarchies and feedback loops – thereby mathematically providing insight into the dynamic behavior and evolution of the system (for information on this mathematical process, see Godet 1986; Gordon & Stover 2003, and Scholz & Tietje 2002). Development and analysis of an impact matrix is inherently participatory, bringing stakeholders and experts through the exercise of brainstorming how key factors interact to drive a phenomenon.
Case study: application of the PS-PEP
Case study context
Jalapa is a rural town of 85,000 people in north Nicaragua. On the periphery of the town center are more than 140 rural communities that rely on surface and ground water sources, using spring catchment, gravity-fed and rope pump systems. Most of these water systems were installed by the Jalapa municipal government or non-governmental organizations (NGOs). The water service scheme in Jalapa is primarily community-based management. The water and sanitation committees (CAPS) are a part of a larger regional network intended to provide training and legal support. Given their intimate knowledge on important factors that influence regional water service sustainability, regional CAPS leaders were invited to the PS-PEP workshops; seven CAPS leaders attended both workshops.
Workshop day 1: factor brainstorming and interaction
Workshop 1 began by introducing participants to the overarching goals of the session: to identify and model the interaction of factors that influence long-term functionality of community-based rural water services in Jalapa. Participants brainstormed factors that could influence long-term water service functionality; 17 factors were identified. These factors were condensed into eight clearly defined key factors: Finances, Communication, Administration, Water Resources, Education & Training, Politics, Appropriate Technology, and Water System Functionality. It was important that the group reach consensus on the definition and meaning of each key factor before moving on to the next step (Newell & Proust 2012).
Next, the group performed an impact analysis of factor influence for each factor. The group identified the influence between each of the factors, and scored the strength of interaction from one factor on the other as either 0, no interaction; 1, weak; 2, moderate; or 3, strong. The influence of Factor A on Factor B (i.e. Factor A → Factor B), then Factor A on Factor C and so on, were performed until all pairwise comparisons were complete. Workshop 1 concluded once each of the influences, and their associated weights, had been discussed.
Model preparation and analysis
Based on the interpretation of the influence map (Figure 4) explained earlier, analysis shows the factors Politics and Education & Training are the most influential on overall sustainability; that is, they are the most influential, and yet least dependent, on the other factors. This implies politics in Jalapa are the leverage point for either good or bad project outcomes. On the other hand, Finances was found to be the most unstable, meaning highly influential yet impermanent in its role as influential or dependent. In other words, adequate finances are necessary to maintain water services yet they would depend greatly on many other factors. Here it is also seen that Communication, Administration and Appropriate Technology exist at the crossroads between influential and uninfluential, implying the potential for either influence or insignificance. Interestingly, the factor Water Resources was low for both influence and dependence, meaning an unimportant driver for sustainability of water services in Jalapa. We revisit this finding later in the case study. Finally, as expected, Water System Functionality is seen to be the most dependent on influence by the other factors.
In addition, it is possible to infer important aspects of factor evolution by comparing direct and indirect influence maps (Figure 4(a) versus 4(b), respectively) to show differences between direct and indirect (i.e. pathways and feedback) factor influences. For example, there appears to be an inferred shift in location (and thus influence) of Administration and Communication over time. Administration shifts in both influence and dependence over time (moving up and right), and Communication shifts in influence (moving up). This implies that water service administration (i.e. management) becomes both more important and more dependent on other factors over time. Influence by communication, however, would increase over time and remain consistent in its overall impact. In addition, Water System Functionality and Finances over time would become more influential on sustainability of water services in Jalapa, yet maintain susceptibility to influence from other factors.
Workshop day 2: discussion of model
Workshop 2 was designed to guide the participants through the discussion of the model findings (presented in the previous section), while allowing time for participants to voice their comments, questions, or concerns with the model outputs and interpretations. The agenda used for the workshop was as follows:
Summary and discussion of the activities in Workshop 1
Summary of the model results and their significance
Group discussion regarding the implications of the results
Group discussion of plans of action.
A summary of the previous workshop activities involved asking participants if there were any observations or key factors that stood out based solely on the pairwise connections made between factors. The summary of model results and their significance began by discussing the final influence graph (Figure 3). The group was then led through an explanation on the visual tools used to analyze factor influence and dependence (Figure 4). To facilitate discussion on model results, the group was asked to consider the relative dependency or influence of each factor asking questions based on the model interpretations presented in the previous section, such as: ‘the model says that education and training of CAPS and community members is the most influential aspect for the sustainability of water projects in Jalapa. Does everyone agree with this? What does this possibly tell us?’ After each factor was considered in this way, the group was asked to discuss a plan of action for current and future water service management in Jalapa.
In the first workshop, the process of considering pairwise factor influences indicated the importance of financial sustainability. A focal point in the conversation was the importance of financial contribution by water service users in support of operation and maintenance of water systems. Participants left the first workshop with a sense that payment of adequate fees by individual water users would be a significant driver of sustainable WASH services in Jalapa.
In the second workshop, participants were presented with the model diagram and the results from the MICMAC analysis. This shifted the focus from the importance of adequate tariff structures to water resource management. As can be seen in the influence maps in Figure 4, the factor Water Resources was found to be both uninfluential and independent from the other factors, and thus uninfluential on the overall sustainability water supply services in Jalapa. Unanimously, the participants rejected this model outcome, where one participant stated, ‘if there is no water, there is nothing, it should be [ranked] first; it is primordial’. When participants realized that the model outcome was based on the designation of influence they had assigned for Water Resources, they adjusted accordingly. As one participant noted, ‘we did not give it [Water Resources] the same importance since it is not looked at with the same understanding that we see now’. Earlier understanding was apparently based on a non-systems view of water service delivery that had changed over the course of the workshops.
The focus on water resources resulted in a discussion regarding deforestation influences on the water table, and the impact of regional and national politics on forestry practices. This bridged the discussion towards the factor Politics. Politics was found to be a highly influential and but not a dependent factor. When asked why this might be, one participant remarked, ‘It is because you have to search them [the government] out. I think it is part of the lack of communication. If you do not ask, they will not know.’ Participants realized that project success was dependent on indirect influences, as one participant commented, ‘[project success] depends a lot on politics – because we have seen a lot of change at the national level … it [politics] will influence water system operation. It is a chain.’ Discussion continued regarding why community CAPS officials tend to refrain from interacting with the municipal government, agreeing that a key reason was a discouragingly slow or non-existent response to community solicitation.
Team understanding on project interconnectedness was realized when the conversation shifted to the factor Communication; as communication seemed to participants as the factor that held politics, water resource management, and community tariffs in tension. Indeed, the influence map (Figure 4) shows Communication as highly influential – indicating a potent driver or barrier to water service sustainability. Digging deeper, one participant commented, ‘It [Communication] is more important than other things, because if they [water committee and community members] lack communication, they cannot manage.’ This observation on the tightly coupled communication–management (Administration) link had a significant effect on the group's understanding of project interdependencies. A participant noticed that Communication influenced service Administration, and that Administration was both influential on project success and dependent on other factors. The conversation ended with one participant summarizing the implications of the factor Administration, ‘administration is influential and dependent – what does this mean for sustainability? Control of [meeting] minutes, management, fund management, operation – it is affected by and influences the other factors.’
An increase in CAPS member understanding on factor influence, dependence, and interdependence pointed the group to a set of action items. The group discussed and agreed upon five categories of strategic action:
Promoting intercommunication between regional CAPS
Scheduling meetings with government authorities
Enforcing environmental laws
Improving communication of problems and solutions
Improving communication between CAPS and community members in general.
While this study presents clear advancements in participant understanding of water service complexity in Jalapa, we note two important caveats for workshop execution related to facilitation and cost. First, due to the relatively specialized and technical nature of the PS-PEP, successful execution is contingent on having a skilled workshop facilitator. Attributes of a skilled facilitator are: a strong understanding of the proper execution of the various steps as well as outputs for the PS-PEP, along with strong interpersonal and communication skills to guide and stimulate participant conversation on factors, interactions, and model implications. Second, while cost is also an essential factor in the scalability of any program, the nature of a single piloted case study does not provide an accurate sense of cost-at-scale. Aside from the specialized facilitation and analysis, the workshop resource requirements would be similar to other workshops of this nature. Further execution of PS-PEP in differing contexts would conceivably refine these aspects of proper facilitation and cost.
By the end of the PS-PEP workshop, participants demonstrated a growth in learning (research question 1), an improved understanding of the problem complexity (research question 2), and identification of action items formed by insight into factor influence, dependence, and interdependence (research question 3). Overall, the PS-PEP appeared to improved participant understanding on the importance and interconnection between factors, thereby facilitating their thoughtful identification of action items that could later serve as leverage points for sustained provision of water services in Jalapa.
We are cognizant that data and interpretation used in the PS-PEP are subjective, and that evidence of the true impact of PS-PSP on water service delivery outcomes in Jalapa remains implicit. Regarding the former, a useful direction for future research would be to explore ways to minimize biases and improve on the internal validity of modeled factor structures, possibly through recurring model building workshops where decision makers modify factor interaction based on updated experience and service outcomes. We prospectively propose this iterative process within Figure 2 as feedback options 1 and 2. With the objective of further assessing the added value of the PS-PEP, we intend to revisit the same CAPS group to investigate and report on the execution and success of any strategic action items that were informed by the workshop. Indeed, the true potential of the PS-PEP may only be realized through subsequent research efforts that execute and evaluate the process for a range of WASH interventions. At the proper scale, however, the benefits of such research and practice could offer advances in knowledge and understanding of WASH service complexity through the dissemination and cross comparison of modeled factor structures and associated outcomes for different regional and WASH sub-sector contexts.
The authors would like to acknowledge members of the Jalapa CAPS group for their time and involvement in this pilot study. This study would not have been possible without their thoughtful engagement within the PS-PEP workshops.